Control, monitor, and log a Spark Core with ThingSpeak

The following web interface was created using javascript plugins on a ThingSpeak channel page to control a LED and monitor the state of digital and analog inputs on a Spark Core using Spark.variable() and the Spark rest api. The idea is to demonstrate a simple way to use ThingSpeak to monitor, control and log data from your core.

The line chart in the channel was updated by using the Spark.client function and by posting to the ThingSpeak API.

The actual circuit is pretty simple:

My goal was to be able to both control and monitor the core using both the Spark and ThingSpeak rest apis with a response time of a few seconds. In addition I wanted to be able to control from a dashboard.

The software is posted on Github and uses a copy of a ThingSpeak library by another author.

The ThingSpeak library was recently updated using suggestions in https://community.spark.io/t/core-becomes-unresponsive-after-http-request-solved/9285/14 to address http caching issues. I copied the ThingSpeak library and renamed it newThingSpeak to avoid a conflict in the Spark IDE. The main code is pretty simple:

    // This #include statement was automatically added by the Spark IDE.
#include "newThingSpeak.h"

newThingSpeakLibrary::newThingSpeak newThingSpeak ("PutYourThingSpeakKeyHere");

    // Define the pins we're going to use
    int led = D0;  // The led is connected here
    int button = D1; // button is connected to D1
    int led2 = D7; // This one is the built-in tiny one to the right of the USB jack
    int analogPercent = 0; // variable to store the ThingSpeak analog value converted to percent
    int analog = 0; // variable to store the analog value
    unsigned long analogUpdate = 16000; // rate in ms to update ThingSpeak
    unsigned long ioUpdate = 900;
    unsigned long lastAnalogTime = 0; 
    unsigned long lastioTime = 0;
    int digitalIn = 0; // variable to store the read value for the button
    
    // This routine runs only once upon reset
    void setup() {
      Serial.begin(9600);
      // Initialize D0 + D7 pin as output
      // It's important you do this here, inside the setup() function rather than outside it or in the loop function.
      delay(2000);
      Serial.println('Starting up ...');
      pinMode(led, OUTPUT);
      pinMode(led2, OUTPUT);
      
      Spark.function("led", ledControl); // define a function to control the led through the Spark api
      
      Spark.variable("AI", &analog, INT); // define a function to read the analog input through the Spark api
      
      Spark.variable("DI", &digitalIn, INT); // define a function to read the digital in through the Spark api
    
      pinMode(button, INPUT_PULLDOWN); // define the button pin as an input
      
    }
    
    // This routine gets called repeatedly, like once every 5-15 milliseconds.
    // Spark firmware interleaves background CPU activity associated with WiFi + Cloud activity with your code. 
    // Make sure none of your code delays or blocks for too long (like more than 5 seconds), or weird things can happen.
    void loop() {
    
          if(millis()-lastAnalogTime >analogUpdate) {
            updateAnalog(); // update ThingSpeak every analogUpdate ms
        } 
        
          if(millis()-lastioTime > ioUpdate) {
            digitalIn = digitalRead(button);       // read the input pin
            analog = analogRead(A0); // read the analog input and return the raw 12-bit value
            lastioTime = millis();
        }     
    
    }
    
    // function to read the analog value and write to ThingSpeak
    void  updateAnalog() {
        
        Serial.println("Loop start");
        
        analogPercent =100*analogRead(A0)/4095;
        bool valSet = newThingSpeak.recordValue(1, String(analogPercent, DEC));
        if(valSet) {
            Serial.println("Value set to field 1 : " + String(analogPercent, DEC));
        } else {
            Serial.println("Value not set successfully");
        }
        
        bool valsSent = newThingSpeak.sendValues();
        if(valsSent) {
            Serial.println("Value successfully sent to thingspeak");
        } else {
            Serial.println("Sending to thingspeak failed");
        }
        lastAnalogTime = millis();
    }
    
    
    // function to control the led through the Spark.io api. Returns 1 if led is on, 0 if led if off, and -1 if the control message was undefined
    int ledControl(String controlString) {
        
        Serial.println("Entered ledControl loop ");    
        
        Serial.println(controlString);
    
        if (controlString == "on")
        {
            digitalWrite(led, HIGH);   // Turn ON the LED pins
            digitalWrite(led2, HIGH);
            return 1;
        }
        else if (controlString == "off")
        {
            digitalWrite(led, LOW);    // Turn OFF the LED pins
            digitalWrite(led2, LOW);
            return 0; 
        }
        else
        {
            return -1;
        }
    
    }

If one comments out the section of the code that posts to ThingSpeak:

      if(millis()-lastAnalogTime >analogUpdate) {
        updateAnalog(); // update ThingSpeak every analogUpdate ms

or alternatively if I post to ThingSpeak only and and do not use the Spark.variable(), things work fine. When the code is run with the http client and spark.variable functions together things go wrong and the following occurs:

  1. The software works as expected some of the time. The ThingSpeak channel updates, one can see the state of the core switch and one can toggle the LED on and off using the button.
  2. The Spark Core reboots randomly
  3. The ThingSpeak write fails randomly
  4. The Spark api calls using the Spark.variable() function fail randomly

Based on what I have read, my suspicion is that this is an http cache and/or timing issue and I am hoping that @bko and @Hootie81 or others might have suggestions on where to start troubleshooting?

4 Likes

I have an event tonight but I will try this out after that.

I tried this out last night and was eventually able to reproduce the failure. I have some idea to try out around the client.println (sends a lot of packets) and all the Arduino String objects (leads to heap fragmentation).

Hi @mawrob

I have sent you a pull-request to make some changes.

The biggest change was calling client.stop() after doing the HTTP GET request. On many Arduino platforms, like WizNet Ethernet boards, there is only one TCP connection possible so calling client.connect over and over just gets you the one and only socket.

But Spark is different and supports many simultaneous connections so calling client.connect gives you a new socket every time. There are only seven sockets in the TI CC3000 and one is used by the cloud so it is easy to run out, plus only a limited number of simultaneous TCP connections are allowed. The TI CC3000 will automatically timeout and close connections but that takes time and holds up resources and is generally not good.

I also tried to make more of the String code as statically allocated as possible using concat rather assignment and I changed the client.println statements so that everything is concatenated into one char array and client.write is called on that. This means that only one packet will be sent by the core since right now on Spark (and Arduino) each character in a client.print or println generates a separate packet. This leads to much better performance, but these changes are optional.

I ran a test overnight and I don’t see any major missing data in the ThingSpeak dashboard–which is really cool to play around with! There are some gaps where one or two of the 16-second samples are missing but it is much improved. Please test and come back if you are still seeing problems!

2 Likes

Hi @bko

Thanks so much for jumping on this post so fast and for your detailed response. I will test out your changes on my setup and report back.

Cheers

1 Like

Hi @bko,

I ran the test for about 24 hours using using your new library and the configuration I described on Github. The update period was set to every 16 seconds. From time to time I opened the ThingSpeak channel dashboard so that the javascript would run and hit the Spark api. That did not appear to influence the successful update rate. The writes to ThingSpeak were logged and I plotted a histogram of the time between adjacent ThingSpeak updates. The really good news is that the Spark was much more stable and I never saw it reboot although I cannot say for sure that it did not.

The results are as follows:

Not perfect but much better. It is still surprising to me that at times the updates appear to fail for one or more update cycles. Not sure why that is the case and would be good to instrument further and understand why but I am not sure how best to do that? The maximum delay between updates was 219 seconds (off the plot axis).

Thanks again for your support!

1 Like

Hi @mawrob

There are lots of ways to debug this further, including turning on the DEBUG switch and writing a bit of code if you build locally which will spit out a fire hose of debug info. Another idea would be keeping counts that you report back in some way–I like to use Spark.publish for this.

In my testing I commented out all the serial print statements and just lit up the D7 little blue LED when it was in the sendValues method. When it fails, the LED flashes very briefly but when it works the LED is on for 1-2 seconds with some amount of variability.

The way you update lastAnalogTime in your sketch captures this variability since you wait until the end of the function to update the next time. So your 16 seconds is not the initiation rate but the minimum spacing between attempts. If you move the setting of lastAnalogTime to the start of the updateAnalog function or even up into the main loop, it would represent the initiation rate instead.

I think that explains some of the variability around the central values in your histogram and why the mode of the first peak is not exactly on 16-seconds.

Another potential source of variability is the use of DNS. Your core will sometimes have to ask the DNS server (usually your router as gateway) what the address of “api.thingspeak.com” is. When the gateway has that address cached, it returns quickly but if not there is more variability there as it asks the next level DNS server for the address.

I don’t think this completely explains the results you are seeing but if you switch for testing to an IP address that will remove this “noise” from your measurements. By the way, don’t use the char array “72.52.4.119”, instead do IPAddress(72,52,4,119).

I know both of my cores that run 24x7 at home go offline once a day when their DHCP lease expires. They go to flashing cyan for 5-10 seconds and then recover and my code picks up where it left off with no ill effects but there could be as much as 30 seconds while my code is not running. If you only had one outlier that might explain it, but you have a bunch of outliers.

Thanks @bko. I will see what I can do to instrument. As a quick test I decided to comment out the Spark.variable calls and only send data to ThingSpeak. I updated the timer to record at the beginning of the analogread. I did not change to a fixed ip:

// This #include statement was automatically added by the Spark IDE.
#include "newThingSpeak.h"

//#define DEBUG

#ifdef DEBUG
 #define DEBUG_PRINT(x)     Serial.print (x)
 #define DEBUG_PRINTDEC(x)     Serial.print (x, DEC)
 #define DEBUG_PRINTLN(x)  Serial.println (x)
#else
 #define DEBUG_PRINT(x)
 #define DEBUG_PRINTDEC(x)
 #define DEBUG_PRINTLN(x) 
#endif

newThingSpeakLibrary::newThingSpeak newThingSpeak ("myKeyHere");

// Define the pins we're going to use
int led = D0;  // The led is connected here
int button = D1; // button is connected to D1
int led2 = D7; // This one is the built-in tiny one to the right of the USB jack
int analogPercent = 0; // variable to store the ThingSpeak analog value converted to percent
int analog = 0; // variable to store the analog value
unsigned long analogUpdate = 16000; // rate in ms to update ThingSpeak
unsigned long ioUpdate = 900;
unsigned long lastAnalogTime = 0; 
unsigned long lastioTime = 0;
int digitalIn = 0; // variable to store the read value for the button

// This routine runs only once upon reset
void setup() {
  Serial.begin(9600);
  // Initialize D0 + D7 pin as output
  // It's important you do this here, inside the setup() function rather than outside it or in the loop function.
  delay(2000);
  DEBUG_PRINTLN('Starting up ...');
  pinMode(led, OUTPUT);
  pinMode(led2, OUTPUT);
  
//  Spark.function("led", ledControl); // define a function to control the led through the Spark api
  
//  Spark.variable("AI", &analog, INT); // define a function to read the analog input through the Spark api

  
//  Spark.variable("DI", &digitalIn, INT); // define a function to read the digital in through the Spark api

  
  pinMode(button, INPUT_PULLDOWN); // define the button pin as an input
  
}

// This routine gets called repeatedly, like once every 5-15 milliseconds.
// Spark firmware interleaves background CPU activity associated with WiFi + Cloud activity with your code. 
// Make sure none of your code delays or blocks for too long (like more than 5 seconds), or weird things can happen.
void loop() {

      if(millis()-lastAnalogTime >analogUpdate) {
         analog = analogRead(A0); // read the analog input and return the raw 12-bit value       
        updateAnalog(); // update ThingSpeak every analogUpdate ms
    } 
    

    
 //     if(millis()-lastioTime > ioUpdate) {
 //       digitalIn = digitalRead(button);       // read the input pin
//        analog = analogRead(A0); // read the analog input and return the raw 12-bit value
//        lastioTime = millis();
//    }     


}

// function to read the analog value and write to ThingSpeak
void  updateAnalog() {
    
    lastAnalogTime = millis();
    
    Serial.println("Loop start");
    
    analogPercent =100*analogRead(A0)/4095;
    bool valSet = newThingSpeak.recordValue(1, String(analogPercent, DEC));
    if(valSet) {
        DEBUG_PRINTLN("Value set to field 1 : " + String(analogPercent, DEC));
    } else {
        DEBUG_PRINTLN("Value not set successfully");
    }
    
    bool valsSent = newThingSpeak.sendValues();
    if(valsSent) {
        DEBUG_PRINTLN("Value successfully sent to thingspeak");
    } else {
        DEBUG_PRINTLN("Sending to thingspeak failed");
    }

}


// function to control the led through the Spark.io api. Returns 1 if led is on, 0 if led if off, and -1 if the control message was undefined


int ledControl(String controlString) {
    
    DEBUG_PRINTLN("Entered ledControl loop ");    
    
    DEBUG_PRINTLN(controlString);

    if (controlString == "on")
    {
        digitalWrite(led, HIGH);   // Turn ON the LED pins
        digitalWrite(led2, HIGH);
        return 1;
    }
    else if (controlString == "off")
    {
        digitalWrite(led, LOW);    // Turn OFF the LED pins
        digitalWrite(led2, LOW);
        return 0; 
    }
    else
    {
        return -1;
    }

}

Too soon to tell but now I am not seeing any gaps now that the Spark.variable has been removed.

… a few minutes later …

Oops - I should not have jumped the gun. I will see what I can do to instrument:

1 Like

You can also try turning the cloud off completely:

http://docs.spark.io/firmware/#spark-disconnect

I like to use a dedicated pin that when pulled high always keeps cloud on so I can still flash over the air.

@bko. To test the up time of the Spark Core I modified the code slightly to send the millis() value to ThingSpeak rather than sending an analog value. Since millis() resets on reboot this seemed a good test to log if the Spark was resetting from time to time. The Spark has stayed up for 22 hours so far so it seems like your library changes have indeed fixed the stability issues. About 10% of the time the write to ThingSpeak fails once and less than about 0.3% of the time two or more times in a row. Still not sure what the root cause of this is yet.

1 Like

@bko Been keeping this test running on my desk just to keep an eye on it. The millis() has not reset for about two days but a moment ago I happened to catch this event:

Not sure what the cyan (did not appear blue) rapid flash is?

So the rapid cyan flash was the classic symptom a huge TI CC3000 problem that we called CFOD where the core could not ARP for a host to make a connection and it got stuck forever in that state. The patches from TI reduced but did not completely eliminate this problem and if you flood your network with ARP packets you can still make the TI part fail. In normal operation, it recovers and works great.

The core flashes cyan rapidly when it cannot make the cloud connection for some reason including ARP problems or if the cloud goes down or gets a major update. If it continues for some time, it eventually reboots. I have an IFTTT recipe send me email when my cores go down and there was some kind of brief event for me anyway at 10:43AM today for instance. So if you are still using the Spark cloud, you might have to compensate for cloud outages in your testing with a “control” in your experiment.

So unfortunately, that’s a really long-winded way of saying, “I don’t know why that happened.” but it hopefully gives you some ideas of why it could happen.

Just wanted to add a postscript that we’ve contributed a library for Photon and Core to to write or read data to or from ThingSpeak, which has MATLAB analytics and visualization.

See the 90 second getting started video

2 Likes

hi sir,
am sucessful to upload the sensor value to thingspeak ,bt how to act on sensor data with thingspeak app ,to control the relay .
pls provide the solution to it
thanks

Hi @praveenreddy

This example on github:

shows how to read the last value posted in a particular channel. You can change the credentials and use your own channel to read the value and then control a relay with a digitalWrite() command.

sir ,am using the lpc2148 board,
mine project is monitoring the temperature and humidity values and upload to the thingspeak, and based on the temperature value stored in the thingspeak (above 40c) ,the motor should get on ,using the thingspeak app such as react &thingshttp.

i have done the uploading part ,which is successfully completed,unable to control the motor fan based on the temperature value stored in thingspeak,using the thingspeak app
plz, give the detail idea,and if possible the code of my project
thanks
praveen