Author Topic: Ethernet interface on SB65EC stops responding  (Read 21478 times)

peter_f

  • Full Member
  • ***
  • Posts: 18
    • View Profile
Ethernet interface on SB65EC stops responding
« on: July 21, 2008, 04:56:48 PM »
Hi,
Just wondering whether anyone has experienced this before.  I have been using my SB65EC for logging/displaying information from my weather station successfully for some months now but recently I have been having issues with the network interface just failing to respond.  The main program loop (based upon v3.06 of the Web Server) is still running and it still logs weather data to the FRAM chip and everything physically looks OK (LED's blinking etc) but you just cannot get to it via the network anymore.  A reset cures the problem but after about a week, it happens again.  The SB65EC is connected to the weather station via RS232.

I have tried various things to isolate the problem but have not got any closer to a solution and am wondering whether I have a faulty board.  The web interface is exposed on the internet through my home router and I thought it was the victim of some sort of attack but I closed that off temporarily and it still failed.  I thought a new version of my code may have been causing it, but I put the old stable version back and it still fails after about a week.  Snooping traffic on the network shows it to be non-existent on the network although network LED's are still flashing.

Any suggestions greatly appreciated.

Peter

This text has been added by admin! For summary, and to help locate this error, please see:
http://forum.modtronix.com/index.php?topic=1003.msg3340#msg3340

« Last Edit: October 23, 2008, 11:19:10 PM by modtro2 »

OmarZ

  • Hero Member
  • *****
  • Posts: 243
    • View Profile
Re: Ethernet interface on SB65EC stops responding
« Reply #1 on: July 21, 2008, 10:03:56 PM »
I had same problems about time run instability: I rember that suing a direct cross cable connection to my PC was ok for weeks, in the LAN with switch/router after few days and random it simples stop to work(board led flashing but ethernet module not). I din't found the problem and i solve in brutal way: after 8h (never crash in such short time period) i force a software reset! :-o it was too bad and actually other projects are stopped because of board instability (for instance swtching supply close to the board cause high instability and unexpected results).

This text has been added by admin! For summary, and to help locate this error, please see:
http://forum.modtronix.com/index.php?topic=1003.msg3340#msg3340
« Last Edit: October 23, 2008, 11:19:52 PM by modtro2 »

stevet

  • Full Member
  • ***
  • Posts: 10
    • View Profile
Re: Ethernet interface on SB65EC stops responding
« Reply #2 on: July 22, 2008, 05:29:34 AM »
I have seen similar symptoms when connected to a switch, but not when connected directly to a PC - I wasn't too concerned about it because I typically don't connect through a switch - but it would sure be nice if someone knew what the problem was so we would not have this limitation.... :|

This text has been added by admin! For summary, and to help locate this error, please see:
http://forum.modtronix.com/index.php?topic=1003.msg3340#msg3340
« Last Edit: October 23, 2008, 11:20:21 PM by modtro2 »

modtro2

  • Administrator
  • Hero Member
  • *****
  • Posts: 564
    • View Profile
Re: Ethernet interface on SB65EC stops responding
« Reply #3 on: July 22, 2008, 02:12:36 PM »
Thanks for this bug report. I would really like to find this bug, and any additional help will be much appreciated. Our online web server does not have this problem, and runs for months without being reset. It is online at http://203.171.96.206/. I can not remember ever having to reset it due to it being locked up. I am however not using the serial port on it. Do all of you use the serial port when this error happens? Our board is connected to a Billion switch.

If possible, you could compile the source code with the debugger switched on. You can than use the Modtronix Embedded Debugger to see if any error conditions occur in the stack. I just released a new version of this software, that fixes some bugs that the serial port had when sending too much data.

I am also currently busy with a major new release of the Modtronix SBC65EC Web Server. This new release incorporates many bug fixes to bugs reported on this forum, and the following new features:
  • Implements 10-bit ADC by default
  • Implements UDP Request commands. Any tag can now be requested via the UDP command port. Just send the %nn tag via the UDP port, and the response will be returned. For example, to read the value of port B5, send %b05 to the UDP command port, and a 1 or 0 will be returned via UDP to indicate the state of this port.
  • Implements configurable serial busses that can be linked together via a web page for. This includes Serial port 1, Serial Port 2, Two UDP ports, and the I2C Master Bus. Via the web page one can for example link Serial port 1 and UDP port 1. After doing this, everything received via serial port 1 will be sent out on UDP port 1, and visa versa.
Most of these new features have already been implemented. Just busy with last one. Might also add more features for serial LCD, like a configurable UDP port that it sends and keypresses to.
« Last Edit: July 22, 2008, 02:24:06 PM by modtro2 »

peter_f

  • Full Member
  • ***
  • Posts: 18
    • View Profile
Re: Ethernet interface on SB65EC stops responding
« Reply #4 on: July 22, 2008, 04:04:24 PM »
I'm not really convinced one way or the other whether it is a bug or not because in the past, the board had been working fine.  It would run for weeks without issues.  I would only reset it if I made a change to the code.

I had made a small change to my code around the time the problems started but I put the old version back when it stopped responding.  Two other things that I have thought of that it may have something to do with are:

1. I have an FRAM chip installed that I know shares I/O ports with the ethernet controller.  I have been careful with my code so as not to overlap FRAM operations and network operations but the issues started some time after I started using the FRAM chip.

2. I had upgraded the MPLAB C Compiler to version 3.20 around that time as well (I can't remember what the previous version was but it was only a minor upgrade) but I'm pretty sure the previous version of my code that I reverted back to was compiled with the older compiler anyway (not 100% sure as I may have recompiled with the new compiler).

I haven't been keeping a record and don't know exactly when it fails but it always seems to last either two or seven days... usually seven.  I had to reset it last Monday evening so if it follows the usual pattern, it will either fail tonight or next Monday.  I'll keep you posted.

Peter

peter_f

  • Full Member
  • ***
  • Posts: 18
    • View Profile
Re: Ethernet interface on SB65EC stops responding
« Reply #5 on: July 22, 2008, 04:18:59 PM »
Just one other thing.  The weather station is connected to the SB65EC via the serial port so if I turn on debugging and use the debugger, it will not be a true test as I will need to use the serial port for debugging.

Peter

modtro2

  • Administrator
  • Hero Member
  • *****
  • Posts: 564
    • View Profile
Re: Ethernet interface on SB65EC stops responding
« Reply #6 on: July 22, 2008, 09:15:12 PM »
Hi Peter

You said you have the FRAM chip assembled. Be sure that the Chip Select pin of the FRAM is not floating. You have to configure the port pins used for the FRAM as an output, and set it to 1 so the FRAM is disabled.

peter_f

  • Full Member
  • ***
  • Posts: 18
    • View Profile
Re: Ethernet interface on SB65EC stops responding
« Reply #7 on: July 23, 2008, 07:43:44 PM »
Yes, the CS for the FRAM (port F7) is configured as an output.  I am using the code by "Ziggurat29" that comes with V3.06 of the Web Server which sets all that up within FRAMInit().

Enabling and Disabling CS is handled with FRAMSPI_AssertCS() and FRAMSPI_NegateCS() which are called within FRAMBeginRead()/FRAMEndRead() and FRAMBeginWrite()/FRAMEndWrite().

Peter

peter_f

  • Full Member
  • ***
  • Posts: 18
    • View Profile
Re: Ethernet interface on SB65EC stops responding
« Reply #8 on: July 29, 2008, 06:15:34 AM »
Hmm, lasted eight days this time... not seven.  It failed sometime in the last couple of hours as I had tested it around then.

I'm sitting here wondering what else I can do to troubleshoot.  It is still dead on the network but to look at, it all looks fine.  Activity LED's are flashing, network LED's are flashing.  I have Wireshark fired up listening on the network for anything coming from that MAC address and it sees nothing (unfortunately, it would only see broadcasts anyway as it is plugged into a switch).  Both Modtronix board and switch are showing link as up.  Unplugging and plugging in the network cable doesn't fix it either, the only way I know how to get it going again is to remove power and plug it back in again.

I'm fairly certain it is nothing to do with the FRAM as while it is dead, it is happily logging weather history to the FRAM chip.  I would expect that if something was wrong there then that would probably stop working as well.

Just tried adding a static ARP entry to my workstation so that packets get directed to that MAC address.  I see the SYN packets being sent on the wire (and network activity LED flash on Modtronix board) but no reply.

peter_f

  • Full Member
  • ***
  • Posts: 18
    • View Profile
Re: Ethernet interface on SB65EC stops responding
« Reply #9 on: August 04, 2008, 05:44:59 AM »
A work colleague of mine has two SBC65EC boards - one of which he is not using much - so he has let me borrow it for a while to see if I can duplicate the problem.  I have soldered on a FRAM chip, loaded my software and web pages onto it and even copied the contents of my FRAM chip to his and now it is up and running exactly like mine.

We'll see how it goes.  Hopefully it works fine, I'd rather just pay out for a new board rather than try and debug what is wrong in the code.   :wink:

sparkcatcher

  • Sr. Member
  • ****
  • Posts: 31
    • View Profile
Re: Ethernet interface on SB65EC stops responding
« Reply #10 on: August 14, 2008, 06:36:38 PM »
peter f and Modtro2:
If you read through this board and the Microchip board I think you'll find enough people who have a similar problem.  The issue is most likely not a hardware problem and most likely a stack (software) problem. 

My sbc45 boards running 3.06 all have the same problem.  I've engineered a fix for my app, but would like to fix the underlying problem. 

Modtro2: I've monitored the stacktsk module using the serial port and logged packet receive errors for over a year and popped the boards with broadcast storms.  Neither are issues.  When the error happens, the receiver module stops receiving packets from the Realtek chip - tcp and udp, nothing comes in even though the receive light blinks on the ethernet connector.  That says the receive buffers have filled up and the packets are being discarded or there is a hangup.  It's very difficult to go any further, you have to monitor the contents of the receive buffers.

It is my belief that somewhere in the stack as all of the housekeeping is being done, something writes faulty data to the receive buffer manager.   It's going to be very hard to find out.  Best way is to isolate by removing modules.  I haven't done this yet.

In the interests of finding the solution,
Peter f:  are you using dhcp?
Modtro2:  it sounds like your board has a static address, but just for research purposes, are you using dhcp?

thanks
sparkcatcher

peter_f

  • Full Member
  • ***
  • Posts: 18
    • View Profile
Re: Ethernet interface on SB65EC stops responding
« Reply #11 on: August 15, 2008, 07:38:06 PM »
Sparkcatcher,
I used to use DHCP (reserved address) but when I started having this issue, I changed to a static IP address as I initially thought it may have had something to do with DHCP lease renewals

Since I have been using my colleagues spare SBC65EC board, I have not had an issue.  It has been running OK for over one and a half weeks using exactly the same code as mine so I was starting to think it is a hardware issue but I'll keep his running a bit longer just to be sure.

The other thing I have been meaning to do is to put the default Web Server V3.06 code back on mine and let it run as well and see if it still suffers the same issue.  I'll try and set that up today.

Cheers,
Peter

peter_f

  • Full Member
  • ***
  • Posts: 18
    • View Profile
Re: Ethernet interface on SB65EC stops responding
« Reply #12 on: September 01, 2008, 04:04:32 PM »
Well, my colleague's board has been running continuously without incident for about a month now so you would think that I have a hardware issue with mine.  But my board has been running without incident with the default Web Server V3.06 code for a few weeks too!  Try and figure that!

I think I'll swap mine back again and see if it starts misbehaving again.

peter_f

  • Full Member
  • ***
  • Posts: 18
    • View Profile
Re: Ethernet interface on SB65EC stops responding
« Reply #13 on: September 09, 2008, 10:32:57 PM »
My colleague's SBC65EC board I borrowed eventually suffered the same problem yesterday.  It ran without a hiccup for about five weeks so there may still be a bug in either my code or the Web Server/TCP stack.

Peter

jtoebes

  • Newbie
  • *
  • Posts: 2
    • View Profile
Re: Ethernet interface on SB65EC stops responding
« Reply #14 on: October 08, 2008, 07:17:14 AM »
I am experiencing the same problem here on not one but two boards (that I bought at different times).

I'm seeing it stop responding sometimes after only a minute and other times after a day or so.  I was looking at a watchdog timer, but the board is still alive, just not responding to TCP requests. 

Configuration:
  Modtronix TCP/IP Stack Version   V2.47
  Modtronix SBC65EC Web Server Version   V3.06
  No serial port connected
  Only using ports A0-A2 as output to control a relay circuit.
  Static IP address, DHCP disabled
  No FRAM, just the basic board with a daughter card that has the resistors/transistors to drive the relays.

Unfortunately I can't actually see the board in this one to determine if the network light is still blinking, but I have put it into a mode where I know it is running code (I have the traffic light sequence through the lights) but it is not responding to web traffic anymore. 

Interestingly enough, another board that I put the same code on is running fine at someone's home - but they haven't seen a lockup at all.

One possible theory:  I am using the free version of the compiler without optimization.  Is it possible that there is a race condition?
« Last Edit: October 08, 2008, 09:22:34 AM by jtoebes »