How does SparkCore maintain connection to cloud API?

Hello
This is my first post here, so little introduction: I’m waiting for my first SparkCore to arrive (in about a month). Professionally I’m sftware developer in Poland tinkering with electronics and microcontrollers (mostly arduino) for hobby.

I would like to know how the SparkCore bootloading firmware keeps connection to cloud API. I’ve read the docs and the public API functionality is perfectly clear to me. What I would like to know is what is the internal mechanism used to pass the request from public API. For example when user calls https://api.spark.io/v1/devices/{DEVICE_ID}/{FUNCTION} the request has to be passed to specific core. It cannot be passed as a regular HTTP request because cores usually are not exposed in public network. So there are two options I can think of: pooling and keeping constant TCP connection to cloud server. Both these options consume much server resurces and I’d like t know how cloud API copes with increasing traffic.

The reason I asked the question is that I’m considering using SparkCore completely without public cloud API. I would create my own simple service for managing spark core activity or wait for open source cloud API to install it on my server (Rasperry Pi :))

PS: I know I could try to reverse engineer the answer form spark firmware but I decided to ask the question here because I belive that there are many advanced users who would like to use sparkcore independently from public cloud API (do not get me wrong: cloud API is really great, but some guys just have to tinker deeper:)) and build their own service for controlling core. As far as I understand it’s perfectly possible but it would require creating custom firmware (not only applicatin.cpp file) and flashing it through USB with dvu-util rather that over the air.

3 Likes

Hi @panjanek,

Great question, happy to answer it as best I can.

No need to reverse engineer anything :slight_smile: but I assume you’re welcome to, the local cloud should be available in the next few weeks (after the CLI), and it won’t be obfuscated or anything, so it should be easy to read and understand the server-half of the secure protocol we’re using / built.

For the sake of extensibility, I was planning on keeping the core-protocol specific code in a node.js package that we’ll continue to use on the Spark cloud, and would be pulled in by the local Cloud. This way anyone using the local Cloud will continue to benefit from updates both from us and the community.

If you’re interested in studying the client-side of the communications protocol, it’s available as a separate firmware library here: https://github.com/spark/core-communication-lib . I also have a node.js version of the client-side protocol for unit tests, and that’s something we might be able to release as well, I’d have to check (I’d want to clean it first :slight_smile: ).

As far as a high-level overview goes, per core, the connection looks something like this:

  1. The core opens a socket to the server, negotiates a random session key using nonces and a unique public/private keypair on the core, and a saved copy of the server’s public key. The public keys are stored locally on either side and are not sent during the handshake, deterring man-in-the-middle attacks.
  2. Server and Core start an AES session, rotating the IV on every message in both directions.
  3. The core maintains the connection with the server with active keepalives about every 15 seconds.

When you send a command to the API, the request to the API is load-balanced, and the command to the core is routed internally to whichever server is currently connected to your core. There the command is transformed into a CoAP message ( http://en.wikipedia.org/wiki/Constrained_Application_Protocol ) that is sent over the secure channel. CoAP is like an extremely byte efficient version of HTTP, and most messages fit inside a single TCP packet.

Hope that helps! – Happy Tinkering :slight_smile:
Thanks,
David

4 Likes

Thanks for thorough answer, that’s exactly what I wanted to know. I’m still amazed that you provide the cloud service for free. If each spark core maintains TCP connection to the cloud with 15s keepalive that could generate a lot of load. Do you use amazon for hosting spark cloud service? That costs money, seriously how do you do it?

As for the internal protocol I can see that a lot of work has been put into making it fast and secure. I’m starting to think that my initial idea of flashing the core with custom firmware and using TCPServer and TCPClient for managing all the communication manually could be unneccessary. Anyway, I’m waiting for opensource edition of cloud service, when will it be available?

Do I understand correctly that if I setup private spark cloud (after the source will be published) and would like to switch my core to the different service, the place in core-firmware to make the change is: https://github.com/spark/core-firmware/blob/master/src/spark_utilities.cpp
in Spark_Connect function, tSocketAddr should point to different IP?

1 Like

Hi @panjanek,

Thanks, I think the cloud service is a pretty good value too :slight_smile:

I’ve been working on a command line tool that will make it easier to switch your core between servers. Right now that means either generating, or swapping a few keys on the core, and either recompiling firmware or flashing a new server ip to your core as well, so it knows where to connect. This can be a bit much for people getting acquainted with the system, so I’d like to make this as easy as possible. One of our scheduled improvements is to allow an overriding ip address to be written in external flash, this way you wouldn’t need to change that address, you could instead just write it over dfu or serial. My hope is to have at least a simple version of the CLI available in the next week or so, and then a version of the local cloud in the next 2-3 weeks.

Otherwise, if you wanted to change it then yes, it would be here : https://github.com/spark/core-firmware/blob/master/src/spark_utilities.cpp#L618-L622

Thanks,
David

1 Like

Hi,@Dave, just two simple questions.

  1. as far as I know, coap is based on UDP, and UDP is unlike the TCP, does it still support a 15 seconds connection?
  2. and when the cloud wants to send a command, it normally just find and reuse the incoming connection from Spark core and sends the command through this short-lived connection, is it ?

Hi @helxsz

Spark uses CoAP over TCP and opens the connection from the core to the cloud so any local firewall or router issues are worked-around. The TCP connection is kept open continuously while you are using the cloud and your core will attempt to re-establish a connection if the connection is closed for any reason.

If your core goes from breathing cyan to flashing cyan, this typically means it is trying to re-establish the connection. This can happen due to many reasons (including that your DHCP lease has expired which happens to me once a day).

5 Likes

Thanks, @bko, it is good to know TCP based coap.

Hi @bko

Is there a mechanism that the core can detect when CoAP over TCP pings start timing out and execute code on the core in response to such condition? At this point one would assume that the only sensible code (since the cloud is unreachable) would be bare metal actions such as pin state change or some higher level SPI or such? If yes what is the mechanism?

Thanks

Hi @bubba198

When a core/photon determines that it is no longer connected to the cloud, it closes the existing socket and opens a new one and retries. When it does this you main LED will start flashing cyan (instead of breathing). In the current firmware it continues to attempt to connect until a timeout and then restart, but the Spark team has been thinking of ways to have callbacks for these types of system events.

This is easy to test by pulling the plug on your router and watching what happens.

2 Likes