Register Bits

Thanks for conducting the experiment on the gate count of level- vs. edge-triggered interrupt. It is nice to know it is really not much additional logic. Since my background during my career has primarily been as a user of hardware (by writing firmware to control it) and not a designer of hardware, I didn’t have the expertise to conduct the experiment.

One thing that you did not specifically make a point of, was that in order to catch back-to-back error packets in an edge-triggered case, pkt_err_stat must go from 1 to 0 and then back to 1 to provide a new edge. Otherwise, a pkt_err_int will only generate an interrupt on the first error packet after a good packet. One way to fix this is to have the packet-checking state machine clear pkt_err_stat when the next packet starts to arrive, and then set it after it has seen enough of the packet to determine that it has an error.

I like how pkt_err_stat stays set or clear after the packet has arrived. When debugging a problem, the ISR might quickly ack the pkt_err_int bit but when the code hangs or we break into the code, we may be left wondering whether or not the last packet was good or not. We can check pkt_err_stat to see what it says.

I don’t think all interrupts can be generalized. I am a firmware developer and I’ve had to jump through hoops in firmware to use edge interrupt when it should have been a level interrupt, and use a level interrupt when it should have been an edge interrupt.

Most error interrupts have to be edge triggered as the error is only an event that needs to be latched in for firmware.

Write sensitive memory should also be edge triggered as in this case firmware only needs to know about the last event, clear the flag and wait for the next event. There is no persistent state associated with this interrupt for it to be a level interrupt.

On the other hand fifo interrupts should be level interrupts. Normal use cases include an interrupt to be sent when the fifo is not empty. Different operations may find the optimal usage for firmware to just read the fifo, operate on the data, and exit the ISR. If there is more data in the fifo, then the processor will kick firmware right back into the ISR. Also this allows a higher priority interrupt to run instead if needed. I once had a use case where this was optimal instead of checking if there was more data in the ISR. Firmware never had to read or write any registers in this ISR. Other fifo use cases include fifo less than half empty, which alerts firmware it can start writing more data into the fifo. I’ve seen this on IO fifos, where on every interrupt firmware would just write the next 16 bytes into the 32 byte fifo, again without wasting time reading or writing registers.

Another thing to be careful of, is not to layer interrupts in a latch scheme, where clearing involves first clearing the interrupt in the source block, and then clearing the interrupt in the interrupt block. There should be no need to duplicate all of those clearing writes at the different blocks that the interrupt is passed through.

While I’m at it, on a slightly different topic, but on the same note as this discussion couldn’t be complete without mentioning write to clear interrupts vs. read to clear interrupts. The read to clear being obviously faster as interrupts are cleared as soon as firmware is notified. However this detracts from the ability of a debugger or logger to read registers without side affects, and because of this I’ve always seen firmware developers select to use write to clear interrupts. The read to clear could probably be beneficial if and only if it could be tightly controlled to just the fast path interrupts. I would still like to always have a secondary register for debugging that wouldn’t clear the interrupt on a read.

This is getting interesting.

@Gary
Thanks for pointing out that the pkt_err_stat needs to be 0 for at least one cycle when a new packet starts. I’ll update the description to reflect this.

Something else that may be useful is to have a readable bit that shows when an interrupt was missed (i.e. not cleared before the next rising edge). Maybe that’s overkill, but the logic wouldn’t be too hard…

@David
Your FIFO example makes a good case for needing level sensitive interrupts. The moral of the story is that the hardware developer should carefully communicate with the firmware developer to understand what type of interrupt is best for which situation.

Another thing to be careful of, is not to layer interrupts in a latch scheme, where clearing involves first clearing the interrupt in the source block, and then clearing the interrupt in the interrupt block. There should be no need to duplicate all of those clearing writes at the different blocks that the interrupt is passed through.

Please elaborate on this. Is it that the firmware should only need to clear the leaf of the interrupt tree, then the clear should propagate to the trunk without having to do more writes?

Regarding write-to-clear interrupts vs. read-to-clear interrupts, I’ve always been scared of read-to-clear due to issues with memory monitors accidentally clearing events, as you mention. There could be a safe way to to have this though… if each block had a debug mode. Assuming the memory monitor reads addresses sequentially, the RTL, when in debug mode, could perhaps predict when it is the monitor reading vs. when it is the firmware. Another way might be to only clear the interrupt on read if the last read was the block’s interrupt pending register, or one of the block’s other interrupts… maybe that’s getting too tricky. It could save a few precious firmware cycles though.

In coming up with the packet error example, I had originally started to think in terms of a status signal that was asserted when a receiver was in-frame (i.e. has locked onto some framing pattern) and de-asserted when out of frame. This was a little over complicated though, since it would be desirable to have interrupt generated on both positive and negative edges — for in-frame and out-of-frame events. Hence, there may be requriements for a register automation tool to support the following types of interrupts:

• active high level sensitive
• active low level sensitive
• rising edge sensitive
• falling edge sensitive
• rising and falling edge sensitive

Thoughts?

Hi Jeremy

What if the DMA controller flags an error interrupt. Firmware would then need to read the DMA registers to determine what this error is and then clear the interrupt in the DMA registers. At this point the ISR should be done.

I was referring to the case where firmware would have to do an additional write to clear into the interrupt controller to clear the DMA interrupt. This shouldn’t be necessary as firmware has already cleared the source of the issue in the DMA register block.

For the read to clear, what about having:
1) interrupt status register (read only, no side affects)
2) interrupt clear register (write only, clears the interrupt on bit set)
3) interrupt read to clear status register ( read only, clears the interrupt on the read)
4) interrupt enable/mask (read/write)

Where (3) could potentially be at the end of the address space for this register block, or is in a separate space where the highest address bit for this block is set.

For example (1), (2), (4) are at 0×000, 0×004, 0×008, and (3) is at 0×100, where this register block uses 9 address bits. In this manner a debug block read could read 0×100 bytes at 0×000 and not worry about hitting the read to clear registers.

It may also be useful in the XML file to indicate side affects on read, in case an automated debug tool is designed to read all registers and output them to a customized debugger.

Just some thoughts. I have no use cases currently for read to clear interrupt registers.

@David
From what I gather, it seems you like the level-triggered interrupt in the case of FIFOs because it allows you to write the ISR so that it exits after processing only one piece of data in the FIFO. Then if more need to be processed, the system will simply pop back into the ISR and handle the next one. You also state that an advantage is that the ISR does not have to read/write any registers. But I can’t see how that is more efficient because when it interrupts back in again, something has to read a register to figure out which ISR occurred and then write to a register to ack it (or at least try to ack it.) Plus you’ve incurred the extra context-switching overhead of exiting and re-entering the ISR for every piece of data. So it seems to me that you could be more efficient if you were to stay within the ISR and loop around for each data.

I don’t quite understand the reasoning of exiting the ISR so that a higher-priority interrupt can run instead. If there is a higher-priority interrupt pending, then, by definition, wouldn’t it interrupt the current ISR in the middle of its execution? Or does your ISR disable interrupts so that the higher-priority interrupt cannot interrupt? If you take advantage of the efficiencies of staying within the ISR and looping on each piece of data, you can put the interrupt disable/restore within the loop, allowing the higher-priority interrupt to occur between data if necessary.

So if the ISR were written to stay within a loop, then level-triggered interrupts have no advantage over edge-triggered, and edge-triggered interrupts can work there just as efficiently.

When you said you like edge-trigged in some cases and level-triggered in other cases, you identified a problem with these variations of interrupts. Because there are variations, not everybody is consistent in which they prefer to be used where. So it generates confusion and frustration. One of the reasons I push for just edge-triggered is that it provides consistency and eliminates that confusion.

I agree that a layered latch is not good. The standard that I advocate is that the non-leaf nodes effectively OR all incoming interrupt lines from the lower-level nodes. Then when the interrupt in the leaf node is cleared, the clearing propagates up the tree.

Yes, read-to-clear does have the advantage of saving one register write. But it cannot be used everywhere, such as in cases where multiple device drivers need to read the same interrupt register to check on just their assigned bit(s). They need to read and ack their own without changing any of the others. Having access to a non-destructive read register does not help because it does not provide a way for one driver to only ack its bits without touching any other interrupts that may be pending. So read-to-clear will not work in all cases, thus forcing some to be write-to-clear. Having some interrupts be read-to-clear and some write-to-clear will cause problems when porting/leveraging code from one to the other. I’ve dealt with that and it made for a very difficult debugging exercise. So from a firmware-quality and a consistency point of view, they should all be the same and should all be write-to-clear.

@Jeremy
If there is a case where multiple edges could occur before firmware has a chance to ack it, then hardware needs to add additional support. Incoming I/O packets is a perfect example of this. In a CAN block that I had to use, there was a counting register that would increment for each incoming CAN packet. When I read a packet out of the FIFO, I would write a 1 to the register which then subtracted that 1 from the current value of the register. Then I would read the register and if it was not 0, I knew there was another packet and would go get it. If it was 0, then I knew I was done. In another situation, in a LaserJet printer, if the block did not have the next strip of page data when the laser needed it, the whole rest of the page was left blank. You can’t stop that whole gear train on a 600 DPI dime. Each strip generates a strip-done interrupt so the next could be loaded. If two strip-done interrupts occurred without firmware loading a new strip in between, an underflow interrupt occurred, thus notifying firmware that two strip-done interrupts occurred. Firmware had to abort the rest of the page and put up an error.

With regards to your question about in and out of frame stuff, I think you are making it too complicated. You already know I’m opposed to level-triggered stuff so that takes out the first two. In my book (which someday will come out) I advocate supporting any needs for interrupting on both edges by splitting it into two separate interrupts, one for the rising edge and one for the falling edge. This allows firmware to have total control over which interrupts to enable, just the rising, just the falling, both, or neither. The same, rising-edge-triggered interrupt module is used. To generate interrupts on the falling edge, simply run the interrupt source signal through an inverter before tying it to the rising-edge-triggered interrupt module. So, of the five types you listed, you only need the third, rising edge sensitive.

The simpler the better, as far as I’m concerned. If a positive edge interrupt can work for each case then that’s ideal.

It’s good to understand the perspective of firmware experts on clear-once-at-leaf interrupts.

@Gary
Coming at this from a RTL perspective — picking up my RTL hat off the shelf where the firmware, webApp software, and business hats are kept — I wonder if “missed edge” detection should be be abstracted into a common interrupt pattern for automatic code generation from a register tool like SpectaReg. The alternative is to hand-code I suppose, but that’s more added work for the RTL developer… and the firmware developer too, since it would be pretty free hand and each RTL code might do it different. I’d be interested to know the readers’ perspectives on this.

Missing-edge-detection logic is only needed when edges can occur without firmware intervention, such as asynchronous incoming data. Most interrupts that I have dealt with cannot happen more than once without firmware doing something, such as a done interrupt when a firmware-initiated task is completed, an error interrupt that causes the block to stop until firmware resets/aborts it and resumes, or an outgoing data transmitted interrupt that requires firmware to give it more data to transmit.

In cases where multiple edges could occur without firmware intervention, the appropriate response will differ depending on the situation. Multiple edges could be ignored, could cause an overflow bit to be set, or could increment a counter. The RTL to hand-code these responses is very small. I’ve seen cases for these three and others. So, not only are there a small portion of interrupts that can have multiple edges, I have not seen a common pattern in what the response to multiple edges should be, so I do not see that having that feature in your tool would be used very much.

Hi Gary

Exiting a interrupt after reading a single ISR, and using the system to pop back into the ISR is more efficient when the probability of a 2nd entry in the fifo is low. This is purely based on a particular use case, and was determined with a lot of measurements on this system. Of course this will vary from FIFO to FIFO within a particular system as well. When I get to the task of determining which FIFO’s fit in which category, I’m really trying to squeeze the extra performance out. Also with a lot of processors, entering an interrupt is very quick with register windows or dedicated ISR shadow registers, so there is only a couple of assembly instructions to the C code.

Interrupts are disabled on the entry to an ISR by hardware. Our firmware does not re-enable them. There are numerous reasons for this, one being that nested interrupts grow the stack quickly. ISR’s should be short lived as well. Quite often an ISR may only check hardware an update a data structure. If the update is not atomic then interrupts would need to be disabled while this happens, and just leaving them disabled during the ISR simplifies the firmware.

What about the situation with 2 fifos. FIFO 1 has a higher priority than FIFO 2. If the FIFO 2 interrupt triggers than firmware will read 1 entry off the FIFO 2 queue. If we had level triggered interrupts, we would exit the ISR right here and let hardware take care of the priorities. If we had edge triggered, when do I clear the interrupt? Based on our priorities, I would only clear the interrupt when FIFO 2 becomes empty. So I read the 1st entry and then determine that the fifo is empty, I can go clear the interrupt. But what if it becomes non-empty right before I clear the interrupt? Now I’ve cleared an interrupt that won’t re-trigger anymore. So instead I have to recheck if the FIFO is empty after clearing the interrupt. If it isn’t empty, we are now in a bit of a situation. So I could check FIFO 1, and process that FIFO until it is empty. Then I know I can read another entry on FIFO 2. Do I clear the FIFO 2 interrupt now? Well I could just stay in the FIFO 2 ISR in a loop continually checking if the FIFO is empty after clearing the interrupt, and call FIFO 1 as needed. This seems a bit more complicated than it needs to be.

Adding my 2 cents input on multiple edge triggered interrupts. I’m not sure how much an automated tool can do to handle all cases. If every interrupt is a DDR ECC error, presumably each master of the memory transaction will stop processing until firmware restarts it, but the DDR controller can only latch in so many errored transactions (usually only 1). In this situation firmware may need to just know that it has occured multiple times and now can only tell what the DDR address of the last (or first) error was, and firmware has no choice and must reset the chip or assert. If the interrupt is an write sensitive memory interrupt, where the memory write is an incrementing counter, then firmware only needs to care about the last interrupt, and doesn’t need to know if there were multiple interrupts before any were cleared.

David,

You have defined a use case with two FIFOs of differing priorities with tight performance requirements and where switching in and out of ISRs is very quick. You have shown that level-triggered interrupts are well-suited for this case. The challenge for me is, can I come up with a way of meeting those requirements with edge-triggered interrupts so that I can consistently use edge-triggered interrupts throughout the system.

For starters, here is pseudo code in a level-triggered case:

isr()
  read 1 FIFO entry
  process entry
  ack FifoIntr // no effect if FIFO not empty

In level-triggered interrupts, firmware has to wait until after the event is cleared (e.g. reading an entry from the FIFO) before it can ack the interrupt, otherwise the acking is of no effect. But in edge-triggered interrupts, firmware should ack the interrupt before clearing the event. Otherwise, it leaves firmware exposed to missing events arriving between the clearing and the acking as you noted. However, acking before clearing does require that the ISR completely empty out the FIFO, not just read one entry. Here is the modified pseudo code for the edge-triggered case:

isr()
  ack FifoIntr
  while FIFO not empty
    read 1 FIFO entry
    process entry

In the most common case of one entry in the FIFO, this adds one new line, “while FIFO not empty.” This line is executed twice, before and after reading the entry in the FIFO. When more than one entry exists in the FIFO, then the ISR will loop as needed to empty it out. If a new entry arrives after the interrupt has been acked but before the looping is finished, then that new entry will be read out of the FIFO but the re-triggered interrupt will not have been acked when the ISR exits. This will cause the ISR to be re-invoked. But when the first check occurs, it will discover the FIFO is empty and immediately exit, which, in your scenario, will be very quick.

It does get a little more complicated when dealing with two FIFOs of differing priorities and you want to allow the higher-priority FIFO (FIFO1) to be serviced before servicing any more on the lower priority FIFO, FIFO2. Here is a possible solution for FIFO2’s ISR.

isr2() // For FIFO2
  ack Fifo2Intr
  while FIFO2 not empty
    if Fifo1Intr pending then call isr1()
    read 1 FIFO2 entry
    process entry

Here is another way of handling this case using a technique I have seen used. It requires support from the interrupt module in the form of another register, PostIntr, that allows firmware to post an interrupt as if hardware had triggered it. After reading an entry, the ISR checks to see if the FIFO is empty. If not, it will post the interrupt then exit the ISR, causing itself to be re-invoked. This pseudo code will work for both FIFO1 and FIFO2.

isr()
  ack FifoIntr
  read 1 FIFO entry
  process entry
  if FIFO not empty then write(PostIntr, FifoIntr)

In this case, only one FIFO entry is processed for each call of the ISR. If there are still more, the ISR will post the interrupt, causing itself to be re-invoked, provided that no other higher-priority interrupt is pending. This will allow FIFO1 to run if FIFO2 is not done yet. It will also allow any other higher-priority interrupt in the system to step in if FIFO1 is not done yet. The PostIntr register allows you to operate as if you had level-triggered interrupts.

What I have shown here, is that it is possible to use edge-triggered interrupts in your example case of dual FIFO with differing priorities. While, from a firmware perspective, it does require a few more lines of code, overall, it reduces overall system complexity and exposure to firmware errors by having all interrupts use the same behavior

Hi Gary

For your level triggered interrupt case you wrote

isr()
read 1 FIFO entry
process entry
ack FifoIntr // no effect if FIFO not empty

What would I ack, if this was a level triggered interrupt? If the interrupt is level triggered, causing the interrupt state to go away de-asserts the interrupt, and there is no ack register.

I would write the ISR as

isr()
read 1 FIFO entry
process entry

Or, depending on the priority of the interrupt and the performance of the system, I would loop in that isr until the fifo was empty.

Gary, I’m not sure why it is an advantage to having one interrupt type for an entire system. In the last 2 chips I’ve worked with, both heavily used FIFO’s for normal data path interrupts, and all other interrupts were signalling error conditions (or unusual operations). This allowed the normal ISRs to process very quickly, as there were no ack/clear registers and optional polling of the fifo not-empty interrupt.

The clock cycle penalty for stalling a processor while waiting for register read data is quite small for simple chips, but can become significant on more complex chips where the read operation may have to pass through multiple bridges/buses.

David

David, when you say…

What would I ack, if this was a level triggered interrupt? If the interrupt is level triggered, causing the interrupt state to go away de-asserts the interrupt, and there is no ack register.

…are you assuming that there is one interrupt signal/pin for the FIFO’s interrupt? Otherwise, if there are multiple interrupts sharing a interrupt pin, there would need to be a memory-mapped interrupt bit that would need to be cleared somehow. Whether level or edge sensitive, I would think the bit would be sticky, remaining asserted until cleared via ack. I suppose the ack could be write-on-to-clear or read-to-clear (the latter having the risk previously discussed).

Can you explain the mechanics of the interrupt such that the SW wouldn’t need to clear/ack it by reading/writing?

Thanks,
Jeremy

David,

What would I ack, if this was a level triggered interrupt? If the interrupt is level triggered, causing the interrupt state to go away de-asserts the interrupt, and there is no ack register.

Don’t you have to ack the processor’s interrupt? It sounds like you have the not-empty status signal feeding directly into one of the processor’s interrupts.

… both [chips] heavily used FIFO’s for normal data path interrupts, … This allowed the normal ISRs to process very quickly, as there were no ack/clear registers and optional polling of the fifo not-empty interrupt.

It sounds like you have an application that needs to quickly move tons of data, such internet network switches. So your design is highly tuned to reduce as many unnecessary register reads and writes as possible.

I am a big proponent of having standard ways of doing things to reduce quality problems. But I also say that if you have a good reason to justify breaking away from the standard, then do it. In your case, in order to fine-tune the system, you have modified the interrupt architecture to allow you to eliminate acking registers in the FIFO’s logic. That style works well in your specific application but should not be used elsewhere without assuring that the circumstances are the same.

Gary

Interrupt system design is not for the faint of heart. Mistakes are easy and they are hard to debug.

There is an inherent problem in level sensitive interrupts. They perform 4 steps:

1) Component asserts interrupt to CPU.
2) CPU does a bus access to clear the interrupt
3) Component removes asserted interrupt
4) CPU resumes normal operation

The problem is that there is a race between steps 3 and 4. If 4 wins then you will get either a spurious interrupt or else multiple interrupt processing for a single interrupt.

30 years ago this wasn’t an issue. You could do a write to clear the interrupt followed by a RTI and everything would work.

With todays processors the write goes into a 6 deep write fifo and waits it turn for the bus. Then it arbitrates with a few dma controllers, goes over a bridge and through a tunnel before reaching it’s destination.

Meanwhile back at the CPU, the service routine is running out
of I-Cache and the hardware stack is in D-Cache and the CPU returns to normal space well before the interrupt has been cleared.

This is one good reason to use a clear on read system. Your program is blocked until the interrupt is cleared.

John, great to hear the case for clear on read! My question, as someone who works with a debugger/monitor, is how do I stop that from messing with my debug flow? The interrupts in the debug environment would not act the same as the full production env since the debugger’s scan of the memory would accidentally clear the interrupts before the firmware sees them. People who use clear on read must have a good solution for this.

Jeremy,

That is why interrupt systems are hard to debug. They are real-time concurrent operations that don’t mesh well with traditional debug techniques. You can make it fast or you can make it debuggable but you can’t do both.

We always put in hardware performance monitors to check the interrupts. Keep a running count of spurious interrupts, If
you see any then figure out why. Measure the service latency and make sure the numbers are reasonable. Keep the design simple and give firmware a lot of flexibility to fix mistakes.

John Eaton

I’m now writing my chapter on interrupts for my book and am writing up this topic on edge- vs. level-triggered interrupts. I came back to this thread to remind myself of the discussion. I have more comments to throw in.

I am going to recant what I said.

The standard that I advocate is that the non-leaf nodes effectively OR all incoming interrupt lines from the lower-level nodes. Then when the interrupt in the leaf node is cleared, the clearing propagates up the tree.

I came across two problems. One is that in some cases, a node might consist of leaf elements and non-leaf elements. In which case, is it an OR or not? The other is that if the tree is big, it forces the handler to traverse the whole tree to find and ack every interrupting leaf node before the interrupt to the CPU is cleared. So by requiring an ack at each node, leaf and non-leaf, the higher level can be acked immediately, freeing up the CPU, and the lower levels can be traversed and serviced as desired.

This also has bearing with John’s comment as to why Read Clear registers are a good idea.

With todays processors the write goes into a 6 deep write fifo and waits it turn for the bus. Then it arbitrates with a few dma controllers, goes over a bridge and through a tunnel before reaching it’s destination.

This architecture requires firmware to wait for that write to go over the hill and through the woods to grandmother’s ack we go until the interrupt clears. The way I have invoked the delay is to follow the write with a read. The read will block firmware until it comes back and by then, we know the write has done its job, the interrupt is clear, and we can exit the ISR.

With the ability to ack at each level and by adjusting the steps, we can solve the clearing problem. When an interrupt occurs, the first step is to read the interrupt register at the top of the tree, at the chip level. That indicates which block in the chip generated the interrupt. The next step is to ack all the interrupts at that top/chip level then follow that by the read of the interrupt register in the interrupting block. By time the read comes back with the details of the block’s interrupt, we know the chip-level ack is complete and the interrupt signal to the CPU is clear. By having to ack at each level, and by acking before drilling down to the next level, we can avoid the step 3 & 4 race condition that John mentioned. Plus this will solve the problem that the Read Clear method solves, therefore we don’t need to implement a Read Clear register thereby avoiding its associated risks.