UDP received DGRAM boundaries lost: read(), parsePacket(), available() all broken

UDP vs TCP (I write this intro to prepare the ground for my description as to why UDP is wrong on the Spark Core.)

TCP is a “connected” protocol. The read and writes happen to a “stream” of bytes which are guaranteed to arrive in order (or not at all - subsequent bytes only arrive if the intermediate bytes arrive - there are no gaps and no re-ordering). The data is not packetised from the application’s POV. (The number of reads and writes may differ, and the write boundaries are not respected. You send 100 bytes and then 300 and then 200, and the other side reads this how they choose, possibly in equal lumps 150, 150, 150, 150.)

UDP is a “connectionless” protocol. The reads and writes happen to individual “datagrams”. The order of arrival is not guaranteed, arrival is not guaranteed. One write creates one datagram. One read reads one datagram. Even if that read is partial, the next read reads another datagram, and not the next one sent, necessarily. The number of reads will be the same as the number of writes [or less if a datagram is lost or more if a datagram is duplicated]. A datagram arrives in its entirety or not at all. (E.g. The sender sends datagrams A, then B, then C each of 100 bytes. The reader chooses to do three reads each limited to 20 bytes. The first read just happens to read datagram C, and because only 20 bytes are read the remaining bytes of that datagram are lost, the second read happens to return the first 20 bytes of datagram A, and datagram B never arrives. The subsequent 80 bytes of A & C are lost too, but by choice - the whole datagram arrives or not at all.)

Why anyone might want to use UDP rather than TCP

On Spark Core I decided to use UDP for the usual reasons. I do not mind if I miss the odd message (i.e. datagram), I do not care if the order of the messages is mangled a little. But I do not want to write the code to work out the boundaries between the messages or to reassemble messages across different reads.

Code demonstrating the issue

In plain old C on a *ix box I created a UDP datagram socket and wrote to it thusly, all the irrelevant code removed:

strcpy( buf, "123");
sockfd=socket(AF_INET,SOCK_DGRAM,0);
for( i=0; i<10; i++) {
    sendto( sockfd, buf, strlen(buf), 0,
             (struct sockaddr *)&servaddr, sizeof(servaddr));
}

On the Spark Core I read the packets like this:

    sbuf[0] = '\0';
    udp.begin(32000);
    for( ; ; ) {
      if( 0 < (packetSize = udp.parsePacket()))
        break;
      delay(1000);
    }
    udp.read( sbuf, 81);
    sbuf[packetSize] = '\0';
    Spark.publish( gsprogname, sbuf);
    udp.stop();

The Spark.publish shows several datagrams are received at once, concatenated wrongly into one. UDP on the Spark Core is fundamentally broken.

UDP is working just fine for me. See Open Energy Monitor port. It has been running continuously for several weeks now. I’m not currently reading the content of the incoming packet but I did originally and it was fine. I found that I had to clear the buffers to 0s before each read and this seems quite common practice. If I don’t do this I get a jumble.

        memset(&UDPoutData[0], 0, sizeof (UDPoutData));
        memset(&UDPinData[0], 0, sizeof (UDPinData));

In this case the buffers are char arrays.

I cannot disagree with what you have written but you have not addressed the issue I report. The issue is that ONE udp.read() must read exactly ONE or ZERO datagrams, not multiple datagrams.

Also, as pointed out below, the Open Energy Monitor port is not UDP, it is TCP.

Let me know whether adding the clear buffer line fixes your problem. If it does then you are only reading one datagram and the problem lies in the string handling not the UDP.

Hi @psb777

I don’t think one read should mean exactly one packet. You seem to be thinking that UDP is a “packetized” protocol like TCP, but it is not. It is a lightweight byte-stream, connectionless protocol where the packet boundaries should not matter. Normally the data sent over a UDP stream is larger than a single packet can hold and so all the received data is just put into one buffer and treated as one stream.

In your example, the packet size is very small and many packets fit into a buffer. If you think of the UDP stream as having 3-byte segments like “123” then at the receive end you need to provide your own framing (figure out where the segments start and end) and read the received data 3-bytes at a time when you are in frame. A lot of time you can just assume that the data is framed and read a segment worth of bytes at a time and process them. As @phec points out, if you re-use your buffer, you might want to clear the buffer.

I think you are thinking about UDP this way because the common Unix implementation of UDP puts one physical packet’s data into one buffer, for reasons of simplicity and performance, but this is not always the case! I have designed an FPGA-based UDP connected device that puts the received data into one big buffer and only uses the physical packets do limited flow control. There are also situations like jumbo ethernet packets where one packet of data corresponds to many buffers since in a typical OS, the buffers are 4k bytes.

@bko No, it is TCP which is NOT a packetised protocol - it is a stream. You have this exactly the wrong way around. Small packets have NO impact on the UDP protocol: If they did then Spark’s UDP would be broken in that way too. There is absolutely NO need to frame messages in UDP, they arrive intact, on their own, or not at all.

How it all works and what you can rely upon TCP vs UDP is as I describe above but I am an idiot, possibly :smile: So I need to find my Stevens networking book. This is the most authoritative I have been able to find on the 1st page of Google results for “TCP vs UDP”: https://en.wikipedia.org/wiki/User_Datagram_Protocol#Comparison_of_UDP_and_TCP

@phec - I have done as you said. I memset the buffer to nul chars now before every read, no difference. Although I felt I was being superstitious doing as you suggest because you will see I use the read count to nul-terminate the string.

@psb777 If you need real networking semantics, you should probably be using something like a raspberry pi instead (unless you’re trying to build a product, in which case you’re screwed). The spark’s capabilities are pretty limited.

Here is the UDP section of RFC 1122 Requirements for Internet Hosts – Communication Layers.

4. TRANSPORT PROTOCOLS

   4.1  USER DATAGRAM PROTOCOL -- UDP

      4.1.1  INTRODUCTION

         The User Datagram Protocol UDP [UDP:1] offers only a minimal
         transport service -- non-guaranteed datagram delivery -- and
         gives applications direct access to the datagram service of the
         IP layer.  UDP is used by applications that do not require the
         level of service of TCP or that wish to use communications
         services (e.g., multicast or broadcast delivery) not available
         from TCP.

         UDP is almost a null protocol; the only services it provides
         over IP are checksumming of data and multiplexing by port
         number.  Therefore, an application program running over UDP
         must deal directly with end-to-end communication problems that
         a connection-oriented protocol would have handled -- e.g.,
         retransmission for reliable delivery, packetization and
         reassembly, flow control, congestion avoidance, etc., when
         these are required.  The fairly complex coupling between IP and
         TCP will be mirrored in the coupling between UDP and many
         applications using UDP.

Notice in particular that UDP only provides checksums and ports numbers. Applications written on top of UDP must deal directly with packetization and reassembly when required.

I don’t think you are using “packetized” in the way that network protocols do. TCP is indeed a packetized protocol and the exact method of packetizing differs for different implementations. In TCP, the TCP packets are the retransmittable entity and so the ACK structure works on the TCP packet level. These TCP packets are carried in IP packets but are not really related to them and TCP packets can be carried over other transports.

I really recommend you read the entire RFC to get a feel for how these protocols work.

@bko, I feel somewhat patronised. This is a complex area but one with which I am fairly familiar over 20+ years of software devt in the Unix environment, some at a fairly low level. I have read some of the RFCs and my copy of Stevens is well thumbed. What you quote doesn’t address the issue which I quite clearly raise. I don’t want reliability. I don’t need to reassemble anything. I just want to rely on the fundamental features of UDP. The datagram arrives, if it arrives, intact and on its own. Your earlier post remains fundamentally wrong, by the way, sorry :frowning:

psb77, I wonder if the problem is with your placement of “udp.begin(32000);” in the loop. This call does a “udp.flush()”, which I suspect will screw up any incoming packets that have not been processed or in the middle of reception. :question:

OK, if Spark UDP is not UDP then it would be nice to know its features. Is there a spec? But maybe I have found a bug with an easy one-liner fix! Would you not prefer that?

@peekay123, if the code I quoted did not feature the loop then the problem would still occur. The actual “reason” I am experiencing the bug I describe (one udp.read returning more than one datagram) is that I delay(1000), allowing for several datagrams to arrive.

Because remove the outside loop, can you not see the Spark code is spinning in the for() loop? And it is while it is there the multiple datagrams arrive from the Unix C program - which has no delays in it.

To make this clear I have gone back and edited the code in my original posting to remove the outside loop. I have re-run the new code for superstitious reasons. No difference.

psb777, I did not describe a bug, I just looked at the core firmware code and common practice. You should move the upd.begin() call outside the loop so it is called once before you send packets. Spark documentation says:

flush(): Discard any bytes that have been written to the client but not yet read.

In your for loop, parsepacket() will look for a complete packet and break out of the loop to process the packet where I can only assume sbuf is sized to a minimum of 81 bytes. You then publish() and close the UPD connection. I would not trust timing of packet arrival based on the delay(1000). You KNOW you are expecting more than one packet (in rapid succession too!) but you are closing the connection! Since UPD requires you to manage the state of the link, then when you listen for a stream of packets that you did NOT initiate, it would seem that you should keep the port open and listening. In an NTP time example, you open the socket, make a request expecting a response (within a reasonable timeframe) THEN close the socket. In your case, the close should come after a reasonable timeout of not receiving packets. All this to say that I believe the issue is with HOW you wrote your Spark code and not on the Spark's implementation of the protocol.

Oh, it also seems that the while() condition will never occur since your for() will loop forever if no packets are received.

Let’s focus on making it work: If you want to read 3-bytes then change this:

udp.read( sbuf, 81);

to this:

udp.read( sbuf, 3);
sbuf[3] = '\0';

Another latent problem is that your code currently reads up to 81 bytes but zero terminates at packetSize, so if packetSize is say 300, the red lights are going to be blinking for you at some point.

@bko I believe the checksum is optional and if not used should be zero so UDP is even closer to null than you suggest. Catching up with your later post - wise words - ideally use fixed length and alternatively use a terminator. The UDP buffer size need not be an issue as long as the variable the data are read into is big enough - you could (very inefficiently) read 3 bytes at a time and process an enormous message as long as you never stored more than you have space for (which doesn’t contradict your advice on the zero at byte[packet size].)
@psb777 I see from bko’s comment why the zeros didn’t work. When I was looking into how to send and receive UDP I gathered that messages might be received more than once or not at all, so regardless of whether your Spark is receiving multiple datagrams you need to check that you have only one message in your buffer. The clear advice on Stack Overflow and the like is that users should preferably use fixed length messages and failing that have an end marker. For my application I now use a fixed length but previously appended a ‘\n’ and read the buffer until I found it.

In other words even if Spark UDP is broken in the sense that it is reading more than one message into the buffer, this is something you should be checking for anyway in case you receive duplicated messages.

Have you checked what the UDP messages look like to another computer? I’ve not duplicated your code but I previously found that I got apparently jumbled messages when reading data on a Pi if I didn’t either fix the length or use a terminator. Here is a very simple python script (that works on a Pi) to see what is going on. This also illustrates the stream like behavior when receiving data.>

#!/usr/bin/python           # Python UDP client

import socket               # Import socket module

s = socket.socket()         # Create a socket object
host = "192.168.1.101"      # Get UDP server name - use your IP
port = 5204                 # Reserve a port for your service.- use your port


s.connect((host, port))
while True:
    datastr = ""
    while True:
        r = s.recv(16)
        text = r.decode("utf-8")
        print(text)
        i = text.find('\n')
        if i == -1:
            datastr += text
        else:
            datastr += text[:i+1]
            break
    print(datastr)
print ("...")

As you see in this case the server uses a newline symbol to mark the end of message. I have used a very small buffer just to illustrate how large messages don’t need large buffers.

1 Like

Yes, that is correct. The checksum (which we don't see on Spark) is optional--you send zero to indicate no checksum and if you are using checksums and it comes out to zero, you send all 1's instead which is treated as the same value in the ring the checksum is computed in.

Port multiplexing is the primary feature and can indeed be the only feature of UDP.

That is also good advice to try Python on another computer.

bko, phec, I love being somewhat quasi-correct :open_mouth: . Thanks for the ongoing education with your posts cause one can never know enough (or too little!) :smile:

@phec : I’m not as familiar with Python as I am with C but I read at https://docs.python.org/2/library/socket.html that the Python socket() is a direct implementation on top of BSD sockets with which I am very familiar. Your Python example is not of UDP datagrams. I see by the same reference that because you don’t specify a type argument to the socket() call that your program creates a SOCK_STREAM connection, not a SOCK_DGRAM socket. In BSD sockets (and therefore in Unix sockets generally and Python specifically and therefore in Microsoft sockets which is based on BSD sockets and therefore etc etc etc everywhere) SOCK_STREAM implies TCP and SOCK_DGRAM implies UDP. Your example is TCP, not UDP. Your program exhibits streaming for that reason. And is not relevant to this discussion, also therefore.

@bko : You suggest a workaround. But my code snippet is much simplified from a real world example - the packets in my real world example are neither all 3 bytes nor (more importantly) are they under my control. They are broadcast UDP packets which are of variable length and have no end of packet markers nor do they have length counters because this is UDP so datagrams arrive in their entirety or not at all and they are never concatenated together. There will be no easy workaround. But the issue is not one of workarounds: My aim was to make the problem with Spark UDP known: It is broken in that one read does not return exactly one (or zero) datagrams.