Recovery procedure for Platinum rel. 2517

Release 2517 of Platinum firmware for CMG-EAMs, CMG-NAMs and CMG-DCMs contains a bug which, under certain circumstances, can cause the file-system to become full, eventually preventing log-in.

This page describes a procedure to recover systems in this state. It involves an upgrade to the latest version of Platinum so, to carry out the procedure, you will need either:

Access to the Internet;
Access to a Platinum Upgrade RSYNC Mirror, preloaded with a more recent version of the firmware; or
A Platinum Upgrade Pod.

The upgrade procedure itself is not described further in this document. Please refer to the instructions in MAN-EAM-0001 for more information on upgrading Platinum systems.

Release(s) affected:	Platinum firmware release 2517
Symptom(s):	Unconditional creation of backfill database files (even when backfill disabled) causing, eventually, a full file-system with consequent problems. Unwanted files are created in the directory `/(null)`
Requirements:	You will need: Physical access to the system; An RS232 console cable; A laptop or PC with a serial connection and a terminal emulator; and The ability to perform a standard upgrade (e.g., Internet access, rsync mirror or upgrade pod).

Procedure:

Disconnect the power supply from the system
Attach a terminal emulator (configured for 38,400 Baud, 8 data bits, no parity, one stop bit, no flow control) to the console port on the 9-way 'D' connector under the lid
Reconnect the power supply to the system
Watch the boot messages. This will take some time (3–5 minutes). When you see the line
Attempt rsync update ? (y/N)
hit the enter key or type an 'N' in order to answer "no" to the question
Answer no to the next three questions until you are presented with a shell prompt.
Enter the following command exactly as given (including the double-quote characters):
rm -rf "/mnt/(null)"
This will take a long time to complete (perhaps 10 minutes, although on one unit we recovered this took over two hours).
Enter the reboot command and then allow the system to boot normally:

/mnt/sbin/reboot -f

Perform a standard upgrade to the latest Platinum release and reboot.

Sample session

The following screen-grab shows a typical session:

Welcome to ARMmonitor running on Compulab CM-X270.
Copyright Compulab 2002-2007 (c).

Built at: Wed Dec 19 22:57:37 IST 2007 on 89
NAND MFG: 0xEC, ID: 0xDC109554


CM-X270 hardware configuration: 
-------------------------------
SDRAM size ............................ 64MB
NOR flash ............................. AMD or compatible, 4MB
NAND flash ............................ Samsung, 512MB
PCI bridge ............................ not present
Ethernet on CORE ...................... present
Ethernet on BASE ...................... not present
AC'97 CODEC ........................... not present
RTC ................................... present


Linux image detected in flash at 0x00040000.
Loading linux kernel from flash... Done.
Booting...
Kernel parameters: 
KERNEL_PARAMS: 0xA0000100

many lines omitted....

yaffs: dev is 32505859 name is "mtdblock3"
yaffs: passed flags ""
yaffs: Attempting MTD mount on 31.3, "mtdblock3"
block 3576 is bad

a long wait after bad blocks are listed....

Attempt rsync update ? (y/N) N
Boot main via kexec ? (Y/n) N
Boot main via switch_root ? (y/N) N
Mount successful but no boot options available
Attempt scripted NAND device reinstall ? (y/N) N
Starting shell. (will attempt switch_root on shell exit)
/ # rm "/mnt/(null)"
/ # reboot

The system was then power-cycled, upgraded and re-booted.