Bug bounty: Kill the 'Cyan flash of death'

@sjunnesson How is the new code running for you?

well not to well, I checked out the firmware, switched branch to “spark_master_new”, compiled and loaded it on 3 Cores and all have the same strange behavior looping over and over with some variation some times in the order it happens. But they never breath Cyan as they normally do when connected to the cloud.

Blinking green
Blinking Red
Quick breathing Cyan

Is this what you expect @david_s5 ? It is with your default application.cpp Just for reference with the default Web IDE firmware all connects to the network so the SSID should be correct.

@david_s5 I just cloned, built and installed your spark_master_new branch and, unfortunately, it’s a blinking rainbow. By that I mean, it only breathes cyan for a few seconds here and there. Most of the time, it’s blinking green and red at various frequencies, as well as fast cyan.

I see @sjunnesson posted similar results while I was writing this.

Are there some build options for debugging?

I remember that it sometimes just began blinking green again and then connected again. But that was sometimes, not always. For most of my tests, I was able to call the function once, then the core would look (RGB LED) completely normal but no function call would work any more - just time outs. Also, because the D7 LED that I toggled, I believe it really stopped.

My latest code is here: https://community.spark.io/t/adafruit-neopixel-library/1143/104?u=hansamann

What is a bit strange is that for @Bdub it seems to work. The only notable difference here seems to be that I have a Kickstarter RGB123 board, 64 xWS2812B while I believe his is an Adafruit 8x8 board. But if I understand the NeoPixels correctly, the spark just sends a signal - e.g. if that is wrong the worst would be the RGBs do not light up. But it should never halt the Spark.

@sjunnesson Oh that sounds not good. Let’s see if we can troubleshoot it.

How about adding DEBUG_BUILD=y to the make command line after a make clean. Do a make DEBUG_BUILD=y

If you are running my version of the application.cpp the debug output will be on tx,rx 3.3v pins.

Does it act the same?

@bkize sorry forgot to include you on last post see above

World Record: over 100.000 secs and still pulsing on “master_new”. Saw it losing the connection and recovering…

Hey all,

Wanted to share a report from Texas Instruments; thanks to @AndyW and @mohit for staying on them like white on rice.

Texas Instruments has realized that this issue is serious, and has sent our support person, Raghavendra, from India (where TI's support staff is based) to Israel (where TI's firmware team is based) to work on this issue with the firmware team. The root cause of CFOD is now on the top of TI's priority list.

Quote from @AndyW:

The bad news: the fix isn't immediately clear, so it's going to take some time for TI to turn this debugging work into a firmware patch. Hopefully the work that many on this thread are doing to protect against failure and ensure the Core and CC3000 recover will act as a stopgap while we wait for the official fix from TI.

Thanks everyone for the great work and support on this issue! We will continue to keep you updated.

6 Likes

@sjunnesson @bkize

I just built the release version with my tools stream CodeSourcery and it runs fine on my core. So the questions is what is the difference?

To that end http://nscdg.com/spark/spark_new_master_core-firmware.zip has 2 builds of my test application.cpp
(as of c5da467fc3cc641ee538208eb819ec43c9cf6d4c ) one debug the other release.

@sjunnesson @bkize would you load these and see it the issue persists?

@Frido What tools and command line did you use?

@BDub and @mohit Would you have time to see what results you get from a build of the branch?

In the mean time I will build an dev environment from the core-firmeware instructions and see if I can figure out the difference.

Thank you all for you help.

David

2 Likes

Just the default toolchain for OSX as described on https://gist.github.com/joegoggins/7763637

ran my shellscript (mentioned earlier), copied the appplication.cpp for my LCD to /src and ran ‘make’ from /build

flashed with dfu-util

nothing special…

Being happy with :spark: up for 115622 sec,

Frido.

basically I followed @Frido description earlier to checkout the repo

rm -rf git_master_new
mkdir git_master_new
cd git_master_new
git clone https://github.com/davids5/core-firmware.git
git clone https://github.com/davids5/core-common-lib.git
git clone https://github.com/spark/core-communication-lib.git
cd core-firmware
git checkout spark_master_new
cd ../core-common-lib
git checkout spark_master_new 

then build and deploy with normal toolchain

make
dfu-util -d 1d50:607f -a 0 -s 0x08005000:leave -D core-firmware.bin

Im using this version of gcc

gcc-arm-none-eabi-4_8-2013q4

@Frido Thank you for the feedback.

FYI @sjunnesson, @bkize, @BDub @mohit

I just finished a build with GNU Tools Arm Embedded (4.8 2013q4) on Win 32 per the core-firmware and I get the read flash.

It is a PANIC out of heap! So the I suspect that CodeSoucery does a better job of coalescing

In release mode the IWDT (wtach dog) is cutting short the SOS and blink codes. :frowning:

In debug the output on tx,rx (3.3V) pin is a <PANIC> char* _sbrk(int) (139):Out Of Heap

0000000001:<DEBUG> int main() (103):Hello from Spark!
0000001994:<DEBUG> int Spark_Connect() (616):sparkSocket Now =-1
0000002000:<DEBUG> int Spark_Disconnect() (654):
0000002004:<DEBUG> set_socket_active_status (810):Sd=0, Status SOCKET_STATUS_ACTIVE
0000002012:<DEBUG> int Spark_Connect() (623):socketed sparkSocket=0
0000002018:<DEBUG> int Spark_Connect() (644):connect
0000002164:<DEBUG> int Spark_Connect() (646):connected connect=0
0000002306:<DEBUG> int Spark_Receive(unsigned char*, int) (366):bytes_received 40
0000002314:<PANIC> char* _sbrk(int) (139):Out Of Heap

Let me see what is causing it.

Found the issue!

FYI @sjunnesson, @bkize, @BDub @mohit @zach @zachary

As indicated here https://community.spark.io/t/bug-bounty-kill-the-cyan-flash-of-death/1322/370

There is a significant memory usage difference when building with the two different tool streams.

The fix for the GNU Tools Arm Embedded (4.8 2013q4) on Win 32 build was to add an #ifdef and #ifdef out out the first UDP object and it’s code in my test app. (So under GCC the test app only does a UDP receive and and HTTP get)

This has been pushed to https://github.com/davids5/core-firmware.git

both the debug and release builds are running on my code using the DFU.

@david_s5 Thanks for the quick turnaround. I’ll test your latest spark_master_new branch later today.

Like @sjunnesson, I’m running the Spark-recommended compiler: arm-none-eabi-gcc 4.8.3 20131129. I’m curious about a better ARM toolset if it’s in the sub-$250 range but that’s another topic.

Just a bit ago, I built the latest master branch from Spark - the last merge was “technobly/spi_fixes” - and it seems very stable. I built that branch earlier this week and it was a blink-fest so I had to revert to the compile-server2 branch.

@zach Great to hear about the progress with TI.

@bkize The spark master has some of my spark_master_new in it so I would expect it to be more stable. Once the Pull Request get in there it should match spark_master_new plus any other great fixes that have been submitted.

thanks for the update @david_s5 The new code flashed nicely and is now loaded on 4 Cores here that I will leave for a week to see how they go. Hopefully when I get back from my trip they all will breath happily and still report to the cloud.

@david_s5 Your updated spark_master_new branch is looking good so far. No issues whatsoever.

Yes, I’ve been monitoring the commits to the Spark core-* repositories. Kudos to you for all of your contributions. The next major firmware release by Spark should be significantly better with all of the contributions from the community plus the many improvements made by the Spark team. Great progress a short timeframe.

@david_s5 I tried to run make via command line but it gives me the following error probably because I’m setup with GNU Tools instead of what your using.

Hi RWB,

The temporary solution to that for Windows 7 can be found on this post - https://community.spark.io/t/error-building-firmware-on-windows-7/1301

You have to replace “rm” and “mkdir” syntax in all the "makefile"s. The makefiles can be found in the “build” folder of all the repositories - core-firmware, core-communication-lib, core-common-lib.

Before:

RM = rm -f
MKDIR = mkdir -p

After:

#RM = rm -f
#MKDIR = mkdir -p
RM = "C:\Program Files (x86)\Git\bin\mkdir.exe" -f
MKDIR = "C:\Program Files (x86)\Git\bin\mkdir.exe" -p

@RWB If you have GitHub for Windows installed - which I highly recommend - see another option here:

Bottom line: Use the "Git Shell" app/shortcut to open a Git-aware command line that recognizes the commands above, instead of opening a "normal" command line that does not.