The Spark-Core needs a state. It should not forget the settings after a reboot.
I would like to have an online key-value store. e.g. to store parameters and configuration properties even after reboots. With the usual data types.
It needs method calls in the firmware, and in Cloud Rest API as well. Showing them in the build/IDE would be great. There should be some defined behaviour in the firmware if the cloud can not be reaching within some seconds.
(PS: When I read the Cloud-Variables description for the first time, I actually thought it would be this feature. )
I love this idea, and it’s something we’ve kicked around a bit. I would like to add this to our feature backlog, but I’d be curious to hear how much storage, how frequently you’d want the cores to be reading/writing, etc. There would be a lot of details to finalize, but I think this could be a very powerful feature.
I would want to pull our fearless CTO @zachary into this discussion, since he’s designed the API and spark communications protocol thus far.
Maybe up to 256 Entries, each Int/Float 4 Byte. And strings up to 2KB, if you can handle. This should be good for 99% of use cases, even with big xml files. And it’s not a lot for a server.
How frequently sync?
I unterstood the feature to be “pull” only, so the firmware has to ask the cloud for data.
To be able to register a callback, when a Property has changed woule be great. Similar to IRQ in Firmware.
I was thinking like 1-2KBi of storage per user, and you can make the most of it by using uint8_t or least with doubles. This should support thousands of users with even a free nano Redis-To-Go addon at heroku just as an example. Maybe pay a small ($5) one time fee for more data storage?
Storage - keep it small, we don’t want to slice out space that is just wasted under some applications. I would suggest we have a max, but make it dynamic, let it grow ad the space us used
Sync - each entry needs a timestamp, or something that would indicate if it was set via the cloud (when the core was offline) or if it was set by the app in the core (and it was offline). The sync should be transparent to the app and part of the firmware / cloud interface.
Classes - maybe more than one spark can have access to the same data repository, so it would be nice to have private vs public storage on the cores
Callbacks - I can understand synchronous vs asynchronous updates, the former being more expensive, this could be a property when the shared element is created?
So - maybe a cloud synched store, with API’s for creating variables with their specific properties (ie - scope, persistence, sync / async, … and then API’s for reading / writing them locally, via the cloud. and both.
Great idea, and I can see a ton of capability. Happy to help in outlining the requirements.
Redis sounds like the perfect solution on the cloud side of things. Small, fast, atomic, simple, and flexible. Being able to implement something using blpop or pub-sub on the Spark side would be pretty epic. Add it to the backlog with a big next to it!
Keep the ideas coming folks! I’ve been skeptical of this one for 2 reasons.
The Core has lots of non-volatile storage for remembering data across reboots, which is much faster than a network round trip.
There are so many providers out there already for this service. I’m not sure how we can add value.
If you want this feature, help convince me. What user experience can make significantly easier/better? Give me some concrete user stories in a world where this feature exists.
A native function in the firmware would make it very easy. It may even allow for easier implementation of "push" notifications or events from the into a queue in the cloud.
I am looking to use the core in agriculture for sensing and automation. I have found tried most forms of wireless for remote sensors and whilst there are some solid solutions out there the cost is prohibitive. Tests have run show I can actually provide WIFI coverage over 80 acres very cost effectively due to the commoditsation of WIFI infrastructure. Given this the spark.core is an ideal candidate for remote nodes.
Power is the main issue, but assuming at some point soon we can turn on/off the WIFI as required powering cores from solar backed li-pol battery will be sufficient. This then leads me to software…
In this scenario I could run a private-cloud when available, however the sensor data needs to be available to the internet for third party review. In field devices need to be kept as simple as possible so interconnect solely with the spark cloud seems simpler. My thoughts is a third part service will use the API to retrieve the stored sensor readings and provide data archive, aggregation and reporting. With the battery powered cores connecting intermittently there is no reliable way to synchronise data collection.
If the spark cloud was to provide a key-value store then the last n-senor observations could be retrieved by the 3rd party using the API without the specific node being currently connected. An alternative would be some form of message-queue that allowed the last n messages from a given node to be retrieved.
There are any number of ways to solve this, it seems to me some form of temporary storage / cache within the spark cloud could provide a simpler solution in the case of cores that are not permanently connected.
Thanks @deancs! I really appreciate the effort you put into writing that up. Everyone who wants this feature please keep posting compelling use cases like this.
Power can be managed pretty well right now. You can turn off the Wi-Fi (or put the Core in deep sleep) for a specified number of seconds with Spark.sleep().
Sensor data like this could be pushed from the Core to a storage provider like TempoDB. has been talking with them about some integration that would make it easier to get data into their system.
I’d still consider having some form of storage in the Spark Cloud if I continue to hear intriguing use cases. Keep them coming.
I really like the idea of a key-value store, and I especially like the idea that it would be shared between Cores. That way you could provide asynchronous communication by getting/setting values in the Cloud.
Any further ideas for implementation would be very much appreciated!
If the Spark Team gets interested in expanding the cloud side of the platform, please take a look into ThingSpeak. Some of the developers are already using thingspeak for storing time series data / key value pairs / location data and using it to interface with external APIs (via ThingHTTP). The ThingSpeak API is open source and the whole system can be used by Spark internally for an enterprise licensing fee. We would love to work closer.
I just saw this on Hacker news. It looks like Dweet would also do a lot of what people are asking for:
Ridiculously simple data sharing for the Internet of Things.
Fast, free and ridiculously simple— it's like Twitter for social machines. If your product, device, machine, gadget or thing can connect to the Internet, it can use dweet.io to easily publish and subscribe to data.
dweet.io doesn't require any setup or sign-up— just publish and go. It's machine-to-machine (M2M) for the Internet Of Things (IOT) the way it was meant to be.
It would be nice if we could use this as an improvement for the Spark.variable() function, which only works when the core is online. Maybe have the spark cloud cache the value of a “spark variable” when it is requested, and return it if it can’t contact the core. Maybe also have something like Spark.push(variableName) to force an update of the online cached value of the variable.
)
Cores, it would open up even further opportunities 
next to it!
into a queue in the cloud.