Write SSE to file?

I am trying to get a spark powered bedside monitor running (it measures heart rate, blood pressure and temperature of patients at the intensive care). The spark will be used to measure everything after which it will be sent out via SSE (spark.publish).

That data needs to be logged (in project description) to a file (preferably excel) together with time and date (timestamp).
The logged data will be used to plot graphs according to that data via C# code (real time graphs, getting updated constantly).

The question is how do I log this data to an excel sheet (or similar)?

Would CSV work? It's simple to write, and it's readable in Excel. It doesn't retain any spreadsheet formatting, but the data is there.

There is a thread on using Google Doc and a script to read Spark.variables().

Is this what you want to do?

Yes CSV would work as long as all values can be taken out seperately (in other words; I can acces them via some code in C#). Also the timestamp is important, but I guess that won’t be much of a problem if the other values can be logged as well.

In reply to bko:

That is pretty much what I am looking for, the downside is that it cannot be logged frequently, which is exactly what I’m going to do (aprox 1-2 times per second).

I don't have any direct C# experience, but I'm pretty sure it can process CSV data. It's a very simple format, so I don't see why it couldn't (especially if you have Excel installed on the same machine and can access the Excel functionality via C#).

I believe you automagically get the timestamp in your published data. A sample line of data from the spark-hq/Temperature feed looks like: data: {"data":"69.125000","ttl":"60","published_at":"2014-03-12T04:58:09.583Z","coreid":"53ff6a065067544826350587"}. However, if you aren't satisfied with that timestamp, you could use your "listener" client to log its own timestamp when the messages are received.

It looks like the Spark.publish() function is rate-limited on the server-side to 60 events per minute per core (so 1 per second). The post was made a little ways down in the Sprint 7: Spark.publish() released! Let’s build a cloud-connected motion detector thread.

Watching the publish data to spark-cli came out just yesterday and would be possible to Pipe it to a csv file if needed.

It has already been done to log Spark.variables()

I believe @dave can easily get this feature available if it’s needed :slight_smile:

Hi @TheHawk1337

As @wgbartley points out, there is rate limiting on Spark.publish(). Maybe you should consider your own UDP protocol instead?

People seem to like node.js for the type of job you want to do converting Spark.publish() events into a a C# readable format.

But if you just want to get Spark.publish() data into a file, which you could massage later into CVS and you have Linux or Mac or cygwin on PC, you do this:

curl --no-buffer --url https://api.spark.io/v1/events/?access_token={your access token}  > foo.txt

The --no-buffer is key since by default curl buffers its output. Replace the {your access token} with your hex access token.

Data looks like this:

event: State
data: {"data":"Current Color Command: auto","ttl":"60","published_at":"2014-03-18T21:43:48.108Z","coreid":"50ff6a065067545632160587"}
event: Uptime
data: {"data":"7:18:48","ttl":"60","published_at":"2014-03-18T21:43:51.862Z","coreid":"50ff6e065067545641560387"}
event: State
data: {"data":"Current Color Command: auto","ttl":"60","published_at":"2014-03-18T21:43:53.126Z","coreid":"50ff6a065067545632160587"}

You could pipe this into your program or a Perl script that turned in to a CSV file easily as well.

I certainly hope you are working on a proof-of-concept prototype design since the Spark core is great, but maybe not quite ready for medical applications.

There's actually a program out there that can directly convert json to csv (json2csv) directly in the command line. However, your mileage may vary on a Windows platform (I'm assuming that since you're using C#). However, with something like Cygwin installed, you might be able to run it through a series of pipes to extract the data you need (curl → grep or awk → disk file → your program).

There's 7 command-line tools for data science that you might find useful (it's where I found json2csv).

Hey @TheHawk1337,

If you’re subscribed to the stream directly, or using the CLI or wherever, you can parse JSON in C# easily with “json.nethttp://james.newtonking.com/json

Thanks!
David

Wow, the help here is amazing :smile:

I should have mentioned that this is for a school project (I study electrical engineering) so no worries about the medical aspect :wink:

@wgbartley The standard timestamp is perfect :slight_smile: Also, 1 publish a second should do just fine :smile:

@kennethlimcp I will look into that, sounds interesting :smile:

@bko I will do some research about that UDP protocol, might just be what I need. Not sure tho. I will also do some research in Perl :smile:

@wgbartley That makes things a lot easier :smile:

@Dave Will look into that :smile:

Thanks a lot all! Much appreciated :smile:
I will do some research and testing and report back after that :smiley:

1 Like