A Silly USB Stick, It Presses 'F1'

So it happened to me. Recently I received some alert that my colocated servers were down at a local datacenter. There had been a power event where the UPS units there had failed causing a reboot of everything in this specific DC. While driving the the DC at 1am to figure out what was going on, I received alerts that some of my servers had came back up, while one had not. Upon arriving there were already a ton of people there dealing with their own equipment. Looking at my specific servers I could see there was a bad DIMM error on the front display, classically flashing orange. "All I need to do is press F1 To Continue" I thought, but unfortunately all of the crash carts were in use by other people there. Anyone who has Dell servers has seen this message during boot at some point if there has ever been any issues with their server. It is a classic error message which I wish Dell had always by default just made the server boot if possible. Because of the queue of people waiting for crash carts, I had to wait until one finally became available to press 'F1' to get my server alive. This could have easily been avoided if I had set this specific server to boot up regardless of error in the bios, but it was too late for this now.

The solution? Well, make sure your servers are set to boot regardless of error in the bios. But a backup solution is my "F1 USB Stick":

What does it do? It is a UDB HID Keyboard that presses F1 repeatedly! That's all!

Simply plug it into a server and it presses F1 for you. No keyboard needed! Your server will be back up in no time.

F1!

I had fun with this one, looking at the kernel messages upon plugging in and you will see this:


usb 3-2: new full speed USB device using uhci_hcd and address 3
usb 3-2: configuration #1 chosen from 1 choice
input: Brad Boegler  Press F1! as /class/input/input3
input,hidraw0: USB HID v1.11 Keyboard [Brad Boegler  Press F1!] on usb-0000:00:1d.1-2

It is based on a Microchip PIC18F14K50 USB enabled microcontroller, a 12Mhz crystal, and the necessary passive components for operation and USB communication. I added a few LEDs in there just to be able to see what was going on. I wrote the code in Microchips C18 which utilizing Microchips USB libraries made this as simple as can be.

I will be keeping one of these in my laptop bag at all times... just in case. ;)

PIC18F C18 Implemented I2C Slave Communication

So after many hours of wasted time I was able to successfully implement an I2C slave device on a PIC18F25K20. What seemed to be a simple task to implement as PIC18F device as an I2C slave ended up being a significant amount of work. This was not because I2C is a complex protocol (it's not), but a combination of attempting to use the built in Microchp C18 compiler I2C libraries for a slave device along with trying to save time by not fully reading the datasheets. This ended up wasting a lot of time. Much searching for the issue turned up with lots of results of people with the same issue, but often as you find in forums, no one had an actual answer to why it wasn't working ( as in no one could get it working! ). My intention here is to clear up this issue and explain what you must do to make a PIC18 work as an I2C slave device. This will not be an I2C tutorial, a good understanding of I2C along with an understanding of the Microchip PIC18 series and C18 libraries will make things more clear.

Using Microchips C18 compiler with the C18 libraries has been very straightforward when using a PIC18F series microcontroller as an I2C master device. If you wanted to write a single byte of data to a slave device with an address of say 0xB0, it can be implemented in the following way:


OpenI2C( MASTER, SLEW_OFF);
SSPADD = 0x27; //SSPADD Baud Register used to calculate I2C clock speed in MASTER mode (in this case 100Khz)

StartI2C();
IdleI2C();
putcI2C( 0xB0 ); //send address
IdleI2C();
putcI2C( databyte ); //send data
IdleI2C();
StopI2C();


This code is simply placing the target device address onto the I2C bus, the upper seven bits of this byte contain the devices address with the lsb bit indicating whether the op will begin a read(0) or a write(1). The data byte is then sent following the address. A standard I2C diagram will explain this the most clearly including the Ack and NAck data.



While still in master mode, reading data from an I2C device can be easily done as well. The following code can be used to read two bytes of data from an I2C slave device from a PIC18 micro operating in master mode:


StartI2C(); // Start condition
IdleI2C();
WriteI2C( 0xB0 ); // addresses the chip with a read bit
IdleI2C();
inbit = ReadI2C(); // read the value from the RTC and store in result
IdleI2C();
AckI2C();
IdleI2C();
inbit2 = ReadI2C(); // read the value from the RTC and store in result
IdleI2C();
NotAckI2C();
IdleI2C();
StopI2C(); // Stop condition I2C on bus


This is also clarified by looking at I2C timing:



Now things become tricky when you want to use two PIC18F devices where one is a master device while the other is a slave. Utilizing the I2C libraries, you would think that implementing something like this on the Slave device would work:


OpenI2C( SLAVE_7, SLEW_OFF);
SSPADD = 0xB0; //SSPADD contains I2C device address in SLAVE mode

while ( !DataRdyI2C() )
{
addr = ReadI2C();
AckI2C();
IdleI2C();
data = ReadI2C();
NAckI2C();
IdleI2C();
}


What the above is attempting to do is wait until the SSPBUF register contains address data, when it does read the byte, acknowledge, wait until the next byte, read that byte, then NAck the data. Of course any data that appears on the bus will not be accepted by the specific PIC at all, so if SSPBUF does ever contain data it will be destined for this device. Another important note is the SSPBUF register will contain the address that is sent upon first byte received. The ReadI2C() function will clear this buffer so even if we have no need for the data, it still must be read.

Now you can play with this code all you want adjusting timing, Ack and NAck sequencing, delays, etc... but ultimately it will not do what you would like it to do / think it should do. Why? Looking into the C18 I2C libraries themselves you will see that some of the functions will only work in MASTER mode, or put simply they were not designed to be used in SLAVE mode. This is where most of my time was wasted.

The solution? Read the datasheet and application notes, one particular app note in particular AN734 will tell you everything you need to know. I would recommend reading AN734 fully if you really want to understand slave communication on a PIC18F. Instead of attempting to modify the C18 I2C predefined functions I decided to implement my own at the register level. Here are the important registers and bits within the PIC18F25K20 (among others) you need to be aware of:


SSPBUF : Serial TX/RX data buffer.
PIR1bits.SSPIF : Interrupt flag bit. This will be 1 when data is received into SSPBUF
SSPSTATbits.BF : SSPBUF buffer full status bit. 1 = full, 0 = empty.
SSPSTATbits.D_A : Data / Address bit. When receiving it indicates what the data received was. 0 = address, 1 = data.


With the above data you can easily implement I2C slave data reception, so here is how. There are two ways you can handle data received. If you code timing is critical, the best and preferred method will be to implement an ISR and place the code within it. If the timing on your device is not as critical you can implement the code in a separate function or within your main loops themselves. Implementation will be up to you.

If you are looking for an interrupt, your ISR will be ran when PIR1bits.SSPIF == 1. Alternately you can look for a few thing to be true while would indicate that a first address byte has been received. Checking for the following will guarantee this case:

if ( PIR1bits.SSPIF == 1 && SSPSTATbits.BF == 1 && SSPSTATbits.D_A == 0 )

With this you are checking to see that an interrupt has been received, SSPBUF is indeed full, and SSPBUF contains an address (not data).
From there you will need to immediately clear the interrupt flag.

PIR1bits.SSPIF = 0;

Then read the byte in SSPBUF

addr = SSPBUF;

Note that the addr byte may not need to be read at all so you can skip this read if not necessary, but be sure to then clear the SSPBUF BF bit. If this is not cleared the next byte sent will cause an SSP overflow resulting in a NAck condition. Reading SSPBUF automatically clears the BF bit.

SSPSTATbits.BF = 0;

Now that the data has been read and/or the BF bit has been cleared you can prepare to receive the data. Depending on the speed of your I2C bus, the speed of your microntrollers, etc the data byte may have already arrived. Before reading SSPBUF blindly as we don't exactly know what is there, we can perform the following check:

if ( PIR1bits.SSPIF == 0 && SSPSTATbits.BF == 0 && SSPSTATbits.D_A == 1 )

This checks to see if an interrupt has been received, SSPBUF is indeed full again, and SSPBUF contains data (not an address). If all checks out we can then read the data byte:

data = SSPBUF;

Now that we have our data we have to do some further housekeeping:

PIR1bits.SSPIF = 0; //clear interrupt
SSPSTATbits.D_A = 0; //set D_A bit to 0 so that we can check if the subsequent byte is more data.

If you wish to receive more than a single byte of data you can then loop through the checks again waiting for each additional byte of data. If you continue to have trouble receiving data from the master, try adding a delay between the first address byte and data byte to be sent on the master. If you are still seeing issues, slow your I2C bus speeds down so you can more clearly see what is going on. A logic analyzer can also instantly show you what is going on with your data. Remember to also watch timing. I was using a 100Khz SCL in the above examples which is slow enough to keep things under control. If you are using a 400Khz SCL (or faster) be sure to enable clock stretching to keep things manageable. Without clock stretching, your ISR may not have enough time to read SSPBUF before the master sends another byte of data.

XBee Based Datacenter Temperature Monitoring Network

Temperature monitoring and control is extremely crucial for any datacenter. I work for a hosting company who has several thousand servers running producing enormous mounts of heat. To battle the BTU output of these servers you must have hundreds of tons of cooling running all the time to keep everything operating under control. Monitoring the datacenters temperature is important in more ways than you think. In the event of a cooling failure, servers will begin to overheat extremely quickly, this in turn causes them to raise the speed of their internal fans to full speed to combat the internal heat. For one server this may not be so bad, but when 1000+ servers do it your total power consumption increases drastically. This can then cause circuits to draw more current which in turn heats them up further increasing the overall temperature in the facility. Servers CPUs will then begin to be throttled by the bios to cool them down which can hurt websites performance. Datacenters internally can actually reach high enough temperatures over 100 degrees Fahrenheit were circuit breakers begin to trip and large 150KVA+ UPS battery backup systems can actually fail from the high temperatures and the increased load they are seeing. Thermal expansion can also come into play causing wire lugs to come loose on UPS inputs and transformers further causing points of failure.

All of this is why reliable temperature monitoring is so crucial. Being able to monitor temperatures throughout a facility can keep you ahead of any potential failure. Commercial temperature monitoring systems are available, but they are extremely expensive and limited on their placement. If you have a 10,000+ square foot facility, monitoring the temperature everywhere could be extremely expensive and difficult to implement. So for all these reasons I wanted to design a system that was very inexpensive and easy to implement anywhere you needed it. Specifically I wanted to not only monitor ambient air in both hot and cold aisles, but also be able to monitor temperatures inside circuit panels and PDUs. Being able to measure a increase in temperature within the circuit panel or PDU itself could indicate a potential failure way ahead of time allowing it to be addressed before it causes a dangerous situation.

My solution was to build a simple temperature monitoring network that was ideal for a datacenter environment. When beginning the design process I came up with the following requirements that it needed to meet:

1.Inexpensive. Commercial temperature monitoring systems are very expensive. I wanted my sensors to be inexpensive enough where you could place them in locations that you would not normally be able to place a temperature sensor because of cost reasons.
2.Wireless. Most existing temperature systems are wired, which is silly in my opinion based on the inexpensiveness of existing XBee based wireless modules. This saves the need of having to run additional wires to each sensor and increases placement flexibility.
3.Easily expandable. I wanted to make sure that I could add additional sensors into the system at any time without hitting any limits on the amount of sensors, within reason.
4.Reliable. Previous systems I used were dumb in the fact that the sensors would sometimes have errors collecting data. This would in turn wake me up at 4am telling me the datacenter was at 150+ degrees Fahrenheit, then instantly back to 70 degrees. This was just annoying.

This is the design of my first three prototype sensors I made:




Each sensor is based on a PIC 18F25K20 microcontroller driven by a 16Mhz oscillator. An XBee module provides the wireless connectivity while a TI TMP100 I2C based temperature sensor monitors the current temperature condition. The TMP100 is one of my favorite chips, it is inexpensive, has 12 bit resolution, and has a wide temperature range. Now I know I could have easily made these much smaller by using all smd based components The PIC I chose is way overkill for the job anyway, but I had a large quantity of these in my parts bin left over from previous designs so I used them as I hate having parts sit around unused.

The software on the PIC simply reads the temperature from the TMP100 every 60 seconds and sends the temp along with a unique device ID to the XBee. The data it sends is also encrypted to prevent anyone from injecting rogue temperature data into my monitoring network. If someone really wanted to they could probably crack the encryption over time. I guess if they just really want to wake me up at night I'll have to implement even stronger encryption. ;)

The receiving end is PHP based. An Xbee / Max232 interface receives the data and sends it to /dev/ttyS0 on the receiving Linux based server. From there I decrypt and parse the received data and process it for errors. It is then checked for predefined temperature limits and emails the appropriate contacts if thresholds are reached. It is also stored into a database and sends a daily log of temperatures throughout the day.

The system is flexible as it allows me to add any additional sensors into the monitoring network at any time. I'm using 2.4Ghz based XBee's, but they are in sockets allowing me to replace them with the 900Mhz version in the event that a facility has a lot of walls blocking the 2.4Ghz signal.

So far the temperature sensors have been working very well, future plans include a modular temperature sensor board and a better (prettier) web interface to show the current facility conditions.

Vehicle Lateral Acceleration Meter

I have often been curious when driving through a sharp turn how much lateral acceleration I am able to reach. Being a Car and Driver subscriber for nearly 15 years now I am well aware of how many G's a vehicle can reach because of centrifugal force. The best vehicles with the best suspension and tires an reach just over 1G of lateral acceleration, so being a spirited driver I want to know what I am able to reach as my tires begin to chirp as I am accelerating out of the apex of a highway cloverleaf at nearly 65 mph.

You can buy inexpensive G meters that you can place on your dash that can show you and log this data along with other fun things like 0-60 times. But these devices are clunky and I hate having things stuck to my dash. In my car there is also no good location to place one. Thinking about this, it would be extremely easy to make one. Accurate accelerometers are dirt cheap, plus wouldn't it be nice to have the data overlaid on my vehicles in-dash navigation system for a nice clean look? I thought so :)

The system is very basic. I'm using a 3-axis accelerometer (only two of the three axis' are in use, the third can be used for acceleration at a later time. Now I am focused on lateral acceleration), a MAX7456 on-screen display generator, and a PIC18F2520 to make it all work.

The PIC utilizes it's 10 bit A/D converters to read in the accelerometer data from an ADXL330 which is good for plus or minus 3G. This data is then manipulated using some basic trig to remove any z-axis rotational data to eliminate false acceleration caused by body roll of the car in a turn. The PIC then sends data to the MAX7456 via i2c to generate a simple bar graph showing relative acceleration between 0G and 1G along with text of the actual current acceleration. The device was easy to calibrate as all you have to do is rotate the accelerometer exactly 90 degrees and get a reading of the earths gravity.

The video output of the MAX7456 will then be overlaid onto my in dash navigation system display. I am using a simple ntsc display for testing on the bench. Looking into the Lexus navigation system in my car I had assumed that the video would be simply composite ntsc, but this was not the case. The video from the nav computer to the in-dash display was RGB destroying my hopes of simply placing the MAX7456 inline with the nav system. Luckily there are inexpensive devices that can plug into the factory wiring harness that can accept composite video and display it on the display. Here is the video output on my bench at rest:

Here is output at 1G (device turned 90 degrees with z-axis compensation disabled):

Here is the device with me shaking it randomly.

Now I just need to get it on the pcb I am in the process of making and get it installed in my car. It has been so cold here in Michigan I haven't had the motivation to get out to the garage and get it going, but should be done soon for it's first test. Once tested the next obvious steps will be to add a datalogger and GPS to track both vehicle speed and acceleration.

Maxim MAX7456 On Screen Display

While looking for a solution to display characters over video for an on screen display I came across the Maxim MAX7456. This is one easy to use chip for exactly this purpose. Using a SPI bus it is easily interfaced with a PIC microcontroller (I'm using a PIC18F2520 for testing).

I have just started playing with this chip and have discovered how easy and powerful it is to use. I have one project in mind for it that I am starting on now and will post again once it is further developed.