“23. However, the Claimants aver that there were a large number of software coding errors, bugs or defects which required fixes to be developed and implemented. There were also data or data packet errors. There was a frequent need for Fujitsu to rebuild branch transaction data from backups, giving rise to the further risk of error being introduced into the branch transaction records. The Claimants understand that Fujitsu maintained a 'Known Error Log' relating to some or all of these issues which was provided to the Defendant but which has not been disclosed. 24. Further, the Claimants aver and rely upon the following: 24.1. Insufficient error repellency in the system (including sufficient prevention, detection, identification and reporting of errors), both at the data entry level and at the data packet or system level (including data processing, effecting and reconciling transactions, and recording the same); 24.1A bugs and/or errors and/or defects in Horizon and any data or data packet errors had the potential to produce apparent shortfalls which did not represent a real loss to the Defendant; 24.2. Horizon is imperfect and has the potential for creating errors (as the Defendant has admitted in pre-action correspondence, in the Letter of Response, dated28 July 2016 , at paragraph 1.3); 24.3. bugs and/or errors have on some occasions produced discrepancies and/or apparent shortfalls (as the Defendant has admitted in pre-action correspondence, in the Letter of Response, Schedule 6) and such shortfalls may also have arisen from data or data packet errors; and, further 24.4. the Defendant sought and/or recovered such alleged shortfalls from Subpostmasters (as is presently understood to be admitted by the Defendant in the Letter of Response, Schedule 6, paragraphs 4.1 to 4.5).” 31. The passages in the Defence which are identified in the Issue are paragraphs 49 to 56. This is a lengthy series of paragraphs but as all are referred to expressly within Issue 1, I will reproduce them all. These state the following: “ Bugs, errors or defects in Horizon 49. As to paragraph 22: (1) If and to the extent that the Claimants wish to assert that any of the shortfalls for which they were held responsible were Horizon-generated shortfalls, it is for them to make that distinct allegation and seek to prove it. Post Office notes that they do not make the allegation in the GPoC. It further notes that, in paragraph 20 of their solicitors' letter to Post Office's solicitors dated27 October 2016 , the Claimants make it clear that they do not allege that there is a systematic flaw in Horizon or indeed any flaw which has caused any Claimant to be wrongly held responsible for any shortfall. (2) It is denied that Post Office has unreasonably or otherwise failed to provide "obviously relevant disclosure" in relation to bugs, errors or defects in Horizon. There has been no order or application for disclosure and, in the premises set out above, there appears to be no basis for providing such disclosure. 50. Paragraph 23 is embarrassing for its lack of particularity, in that (amongst other things) it does not identify the errors, bugs or defects on which the Claimants rely or how "large" their number was or the period in which they are said to have occurred and nor does it identify the transaction data that Fujitsu is alleged to have rebuilt, how "frequent" was the need to rebuild it or the extent of the ''risk of error" which is said to have been introduced. In the premises, Post Office cannot plead to the first three sentences of this paragraph. However: (1) All IT systems experience software coding errors or bugs which require fixes to be developed and implemented. As is noted in paragraph 53 and 54 below, there are robust measures in place in Horizon for their detection, correction and remediation. (2) All IT systems involving the transmission of data over the internet experience data or data packet errors during transmission and such systems routinely have protective measures in place to prevent such errors creating any difference between the data transmitted and the data received and retained by the recipient. Horizon has robust controls making it extremely unlikely that transaction data input in a branch would be corrupted when being transferred to, and stored in, Post Office's data centre in a manner that would not be detected and remedied. (3) Like all IT systems, Horizon has backups to guard against any loss of data due to local hardware failure. Where hardware fails, the data on that hardware is recovered from the backup. Post Office takes the term "rebuild" to refer to the situation before the introduction of Horizon Online where a new terminal was introduced to a branch and the data stored on the other branch terminals (or on a disc where it was a single counter branch) was restored to the new terminal. In this context, Post Office does not accept that there was a "frequent" need to "rebuild" data from back-ups. (4) It is admitted that Fujitsu maintain a "Known Error Log". This is not used by Post Office and nor is it in Post Office's control. To the best of Post Office's information and belief, the Known Error Log is a knowledge base document used by Fujitsu which explains how to deal with, or work around, minor issues that can sometimes arise in Horizon for which (often because of their triviality) system-wide fixes have not been developed and implemented. It is not a record of software coding errors or bugs for which system-wide fixes have been developed and implemented. To the best of Post Office's knowledge and belief, there is no issue in the Known Error Log that could affect the accuracy of a branch's accounts or the secure transmission and storage of transaction data. 51. In paragraph 24, the Claimants again bundle many ambiguous and/ or misleading allegations together. Post Office separates out and addresses those allegations in paragraphs 52 to 56 below. 52. As paragraph 24.1 does not explain what is meant by "error repellency", what sorts of errors are referred to, what is meant by "data entry level", what would constitute "sufficient" prevention, detection, identification or reporting of these errors, or in what respects the error repellency of Horizon was insufficient, Post Office cannot plead to this paragraph. However, the general thrust of paragraph 24.1 is denied and the robust controls, procedures and practices pleaded in paragraphs 53 and 54 below are noted. 53. As to paragraph 24.1A, it is a truism that errors or bugs in an IT system and data or data packet errors have the potential to create errors in the data held in that system. However, Horizon has at all material times included technical control measures to reduce to an extremely low level the risk of an error in the transmission, replication and storage of the transaction record data. These have varied from time to time and they currently include the following: (1) Horizon creates, transmits and stores trans action data in the form of "baskets". A basket is a complete transactional session between a customer and Post Office and may include one, several or many individual transactions taking place within the same session. Horizon will not accept a basket of transactions that does not net to zero (i.e. the value of any sales is set off by the value of any payment made or received). This reduces greatly the risk of any error in the data entered within any given basket. (2) If a basket of transactions fails properly to complete its transmission to the central database (because, for example, of a power loss), the system rejects any partial transmission and requests the full basket from the branch terminal. This reduces greatly the possibility of baskets of transactions failing to be recorded. (3) At the point of a basket being accepted by Horizon, it is assigned a unique sequential number (a "JSN") that allows it to be identified relative to the other baskets transmitted by that branch. This reduces greatly the risk of recording duplicate baskets or there being a missing basket. (4) Each basket is also given a digital signature, i.e. a unique code calculated by using industry standard cryptography. If the data in the basket were to change after the digital signature was generated, this would be apparent upon checking the digital signature. (5) Initial data integrity checks are undertaken when baskets are received at the Post Office data centre from a branch. Baskets are then copied from the central database to the Audit Store where a digital seal is then applied (the "Audit Store Seal"). If the baskets and/ or the data within the baskets were altered after the application of the Audit Store Seal, this would be apparent when the baskets are extracted from the Audit Store. (6) Horizon and the above controls are themselves subject to various audits and checks including audits carried out by third parties. 54. Further as to paragraph 24.1A, in addition to the technical controls referred to above, there are several operational procedures and practices conducted by Post Office and Subpostmasters that serve to increase the reliability of the data stored in the central data centre as an accurate record of the transactions entered on branch terminals. These currently include the following: (1) For many transaction types, Post Office compares its own transaction record against the corresponding records held by Post Office clients. If an error in Horizon were to result in the corruption of transaction data, this should be revealed by the comparison. (2) There are detailed procedures in place to address the risk of data loss resulting from interrupted sessions, power outages or telecommunications failures in branches. These are set out in the "Recovery - Horizon Online Quick Reference Guide" and Horizon guides the system user through the recovery process (which include completing any transactions that are cut short). These procedures should prevent any data errors arising from interrupted sessions, power outages and telecommunications failures. (3) The display of the transactions being effected on-screen at the branch terminal allows the user of the system to identify any inconsistency between the information shown on the screen and the transaction that the user has keyed into the system. If, for example, a hypothetical bug in the terminal were to cause a key-strike on number S to be recorded as an input of number 6, this would be detected rapidly by system users, given the large number of system users and the huge number of transactions effected on Horizon. (4) The accounting and record-keeping obligations placed on Subpostmasters reduce the risk of any errors going undetected. For example, there is an obligation for each branch to count and declare to Post Office the cash it holds on a daily basis, which increases the likelihood of promptly detecting any overstatement or understatement of the cash position on Horizon. If a Subpostmaster detects that an error has been made at an early stage, its cause is more likely to be identified. (5) Fujitsu operates industry standard processes for developing and updating Horizon and for investigating and resolving any identified potential system errors. 55. As to paragraph 24.2, Post Office admits that, like all other IT systems, Horizon is not a perfect system which has never had any errors or bugs. However, as indicated in paragraphs 53 and 54 above, it has robust systems in place to identify them, fix them and correct their consequences (if any). 56. As to paragraphs 24.3 and 24.4: (1) There have been occasions on which bugs or errors in Horizon have resulted in discrepancies and thus shortfalls or net gains in some branch accounts, as outlined in Schedule 6 of the Letter of Response. It is denied (if it be alleged) that such bugs or errors have affected any of the Claimants. (2) On each occasion, both the bugs or errors and the resulting discrepancies in the relevant branch accounts were corrected. Post Office took steps to ensure that it had identified all branches affected by the bugs or errors and that no Subpostmaster was ultimately held responsible for any resultant shortfalls. (3) Paragraphs 4.1 to 4.5 of Schedule 6 to the Letter of Response relate to the so-called Suspense Account Bug. Without prejudice to the burden of proof, none of the branches affected by the Suspense Account Bug are branches for which the Claimants were responsible. (4) None of the Subpostmasters whose branches were affected by the Suspense Account Bug were ultimately held responsible for the shortfalls that it generated. The Claimants are therefore wrong to understand Post Office as having admitted that it "recovered such alleged shortfalls from Subpostmasters". Where Subpostmasters in the affected branches had made good or settled centrally shortfalls that were later corrected, those Subpostmasters received a payment or credit in the amount of the shortfall.” 32. It can be seen that the word “robust” is used throughout these passages of the Defence, and “robust systems” in paragraph 55. Given “robustness” forms part of Issue 3 there is no need separately to consider it at this stage when analysing the differences between the parties about what Issue 1 means. 33. The passages in both parties’ pleadings which I have reproduced above show that it is the potential, in general terms, of the Horizon system to have the effect complained of that is the subject matter of Horizon Issue 1. In order to read Issue 1 in the manner contended for by the Post Office, one would have to delete entirely the phrase “to have the potential” from the issue. One would also have to substitute the term “Subpostmasters’ branch accounts or transactions” with “the claimants’ branch accounts or transactions”
“In Post Office’s submission, the meaning of robustness is a matter for expert opinion. Robustness is a well-established concept in the IT industry and is the subject of academic study: see para. 361 of Post Office’s written closing”. 40. I do not consider that the meaning of words is a matter for expert opinion. The two experts in this case are IT experts, not experts in linguistics or the meaning of language. However, the meaning of robustness within the field of IT is, arguably, a matter upon which the experts’ opinions should be considered, not least because they were applying that term to their expert exercise. The Post Office also relied upon the 1 st Experts’ Joint Statement which in respect of Issue 3 stated the following as agreed: “There are different dimensions of robustness, such as robustness against hardware failure, software defects and user error. The robustness of the system also depends on the processes around it. Robustness does not mean perfection; but that the consequences of imperfection must be managed appropriately. If the extent of imperfection is too high, this would be very difficult to do which would imply less robustness. Horizon has evolved since its inception. Therefore, its robustness may have varied throughout its lifetime. The level of robustness may have increased or decreased as the system was changed. The existence of branch shortfalls is agreed. The experts do not agree at this point as to whether this indicates any lack of robustness.” 41. In the areas of disagreement in this Joint Statement, each expert provided the following. Mr Coyne stated (inter alia): “For the purposes of addressing the robustness of Horizon, I have applied the following definition of robustness: ‘The ability to withstand or overcome adverse conditions, namely, the ability of a system to perform correctly in any scenario, including where invalid inputs are introduced, with effective error handling.’ ”
“The definition of 'robust' proposed above by Mr Coyne is not adequate, for reasons given below. The term 'robust' is not, as implied in para 3.1 of the outline, either ill-defined or a piece of IT public relations. Robustness (which is closely related to resilience) is an engineering objective, and large parts of project budgets are devoted to achieving it. It receives its meaning in the phrase 'robust against... [some risk or threat]', and there are a large number of risks that business IT systems need to be robust against - such as hardware failures, communications failures, power cuts, disasters, user errors or fraud. These are the dimensions of robustness. In all these dimensions, robustness does not mean 'be perfect'; it means 'address the risks of being imperfect'. The extent of robustness is to be interpreted as: in how many dimensions was Horizon robust? and: in each dimension, how large were the remaining risks? In my report I shall survey the evidence I have found that Fujitsu paid sufficient attention to the dimensions of robustness, and that they did so successfully. I shall also address evidence from Mr Coyne implying that Horizon fell short of its robustness objectives. In my current preliminary opinion, Horizon is a highly robust system, and this has important implications for the other Horizon issues, notably issue 1.” 43. It can be seen therefore that Dr Worden in the Joint Statement did not agree Mr Coyne’s definition, and expressly said it was not adequate. In any event, the meaning of any word – even “robust”, or “robustness” – ought to be capable of description by the parties themselves. Although on its face it did not appear that Dr Worden agreed with Mr Coyne’s definition, a footnote in the Post Office first set of post-hearing submissions suggested that Dr Worden was not disagreeing with the first part of Mr Coyne’s text, in other words that part of the text that contained his definition of robustness (which was in italics in the 1 st Joint Statement). Obviously if the parties (or their experts) could agree the definition to be applied so far as the Horizon System is concerned, that ought to be identified. I therefore asked the Post Office via email whether it agreed with the definition adopted by Mr Coyne, and if not, what its alternative definition was. 44. This led to a further document being received from the Post Office dated18 July 2019 . It referred to the passage in the 1 st Joint Statement (which is quoted at [40] above) as “the agreed definition”
“Irrespective of how you define the detail of robustness, in line with most other large-scale computer systems, Horizon's robustness has generally improved. From our experience of other computer systems, Horizon is relatively robust. We agree that 'robust' does not mean infallible and therefore Horizon has and will continue to suffer faults. Robustness limits the impact of those faults and other adverse events . This increase in robustness has, in part, developed from Post Office discovering bugs/errors and defects in live use and then applying fixes and improving monitoring.”
“the robustness of a system is the effectiveness of the system in managing the risks of imperfections (which are inevitable in any system) and their consequences”
“Robustness (which is closely related to resilience) is an engineering objective, and large parts of project budgets are devoted to achieving it. It receives its meaning in the phrase 'robust against... [some risk or threat]', and there are a large number of risks that business IT systems need to be robust against - such as hardware failures, communications failures, power cuts, disasters, user errors or fraud. These are the dimensions of robustness. In all these dimensions, robustness does not mean 'be perfect'; it means 'address the risks of being imperfect'. The extent of robustness is to be interpreted as: in how many dimensions was Horizon robust? and: in each dimension, how large were the remaining risks?” 51. The Post Office also submitted that Mr Coyne’s definition was not “materially different” to that of Dr Worden. 52. The Post Office made submissions in paragraph 3(b) of the written submissions dated18 July 2019 on robustness that stated that Mr Coyne cannot have intended to exclude the effect of countermeasures when he considered the concept of robustness. I shall return to this topic when dealing with countermeasures. This is because some of the countermeasures considered by Dr Worden are not parts of the Horizon System at all, such as SPMs noticing adverse entries in their branch accounts, and the manual issuing of Transaction Corrections (TCs) by the Post Office (which both parties agree are outside of the Horizon System). 53. I do however accept the Post Office’s submissions that there is not a great or material difference in the definitions of robustness adopted by the parties’ experts. I do not accept the claimants’ submission that robustness is difficult to define. Dr Worden defined robustness by using what he termed as “the dimensions of robustness”
“A. I'm experienced -- I have been running a post office for 17 years, sir. I have also worked for the Post Office on training other offices how to run a post office. I was also involved in running and introducing the new Horizon software changes in 2006 onwards, where I went to several offices on behalf of the Post Office to give them training. So I'm an experienced, trained subpostmaster and I ran my business successfully for 17 years. So I may have been a bit brief in the statement but obviously I can run through those -- exactly those steps that we would take to make sure that there is no operator error on our behalf.” 90. The conflicting evidence on this particular point is a good illustration of the “poles apart” position to Horizon by the Post Office and the claimant SPMs in this litigation. Because of the Post Office’s position on Horizon, almost all and any of the criticisms or accounts of factual events which the claimants made, or make, about how this system worked in practice are attributed to fault or carelessness by the SPM or their assistants. Indeed, without fault or carelessness by an SPM, the Post Office simply cannot explain these occurrences. The Post Office’s position is therefore to challenge the factual account – which it is entitled to do – because if the factual account by an SPM is accepted as truthful and accurate, then the Post Office would have to accept that there must be a fault or faults within Horizon. Therefore, the Post Office cannot accept that the factual account is truthful and/or accurate. Thus the dispute goes around and around in endless circles. This litigation is aimed at breaking that deadlock. 91. It is also the case that Mr Latif had the following point positively put to him about why he had checked the CCTV that he had within his branch. “So you now say you looked at the CCTV because your colleagues were concerned that you hadn't done the transaction properly?” even though Mr Latif had said no such thing. Mr Latif was subject to a fairly robust attack, not only on his account, how it matched up with other records which the Post Office said contradicted it, what he and his assistants had or had not been doing, and indeed upon the full scope of his evidence and his credibility – as shown by the question I have reproduced. That question was framed as though even his own colleagues had concerns about what he had done. It was positively put to him by the Post Office that he had not even complained to the Post Office, although he provided the name of his Area Manager Mr Navjot Jando and said he had complained to him many times. The Post Office did not call Mr Jando to rebut this. One exchange will suffice as an example of the type of attack upon Mr Latif: “Q. You don't say anywhere in your witness statement that the£2,000 physical cash also somehow disappeared, but that seems to be what you are now saying, is that right? A. Well, the system gave a shortfall of£2,000 and that's been my statement all the way through, sir, so I don't know what you're trying to confuse me, but there's a shortfall of£2,000 in stock unit AA and there should not be a stock shortfall. The money is physically there.” 92. Mr Latif’s evidence had never been, so far as his witness statement and evidence orally before the court, that£2,000 in cash had physically disappeared. His evidence was that there was a shortfall of that amount shown in Horizon as a result of what he had done. There were some aspects of the cross-examination of Mr Latif which were simply unhelpful. Firstly, extensive spreadsheets were put to Mr Latif which he had not seen before. They are plainly not in chronological order, and had in any event been what was called “filtered” by the Post Office legal team; they were not agreed by the claimants, nor was the “filtering” process explained at any point. There is a limitation on the degree of assistance to be obtained by such an exercise. Further, there was no agreed exercise by the experts whereby the two of them had gone through all the records directly relating to this specific instance and agreed what the records did, or did not, show. The Post Office’s case amounted, literally, to a pure challenge of fact that what Mr Latif said had occurred, simply did not occur in fact. “Q. On the basis of that, Mr Latif, Post Office says there was no failed transfer such as that described in your witness statement and that you are simply wrong about that, it never happened. A. So you are calling me a liar? Q. Mr Latif, you may be mistaken or you may be lying. I put the question that it didn't happen. A. Well, I state that they did.” 93. So far as the other issue experienced by Mr Latif in January 2018 was concerned, the Post Office accepted, as Ms Van Den Bogerd’s original written evidence (prior to later correction) put it, that its data showed “ that the branch received two TAs (meaning Transaction Acknowledgement) on18 January 2018 . However, due to an error by Post Office, instead of increasing the scratch-card stock, the TAs decreased the stock. To be clear, this was a data entry error by Post Office and not an issue with Horizon. Horizon processed the TAs accurately. I note that the TAs were accepted by the branch, which could have been challenged at that point if the user had noticed that the TAs were not for a positive number, as they should have been .”
“A. Can I just confirm, there was an audit done in September of this year, an audit by a Post Office trained auditor, and my stockholding was still showing negative. And a Jane Lawrence is the auditor and she has – still could not resolve this matter, so the problem hasn't gone away, the problem is still there. And there have been a number of calls to the helpline to resolve that negative stock and it hasn't worked. They haven't come back with a response.”
“A. I still state that an independent audit was done by the Post Office, a Mrs Jane Lawrence, in September 2018 the stockholding was still negative and as the branch was handed over to another subpostmaster that is going to be investigated, it's going to be investigated. If you are now coming back with this evidence, I still say that there is a problem somewhere and I don't know what's happened but we have still got a negative stock figure within our branch.”
“KB (Mr Bridges) I can confirm that there are no issues with Horizon and as my letter stated there is no issue with the production of labels. Jay (Mr Tank) Then obviously you conducted your investigation into my concerns without asking me for my evidence. I've got the evidence here to show you. I'm happy for you to take copies but I'm going to hang on to it but how can you say something is fully investigated if you've not looked at all of the evidence.”
“MR HENDERSON: So what appears to have happened was a transaction from the Post Office card account was in the middle of being processed and so it was in the stack presumably. A. Yes. Q. But had not yet been posted to Horizon. So you hadn't cashed out on that transaction, you hadn't completed everything to do with that transaction? A. On the stack – Q. It's on the stack. A. It's on the stack, but the stack has a balance of zero, so to clear the stack you just press "enter" and it goes straight – Q. But you hadn't got to the point of clearing the stack? A. I'm not sure. Q. Okay. My suggestion is that there was probably an outage at just the point where the money had been taken from Post Office card account but had not been processed onto Horizon. That's my suggestion to you. A. Okay. Mr Justice Fraser: Well, is the witness going to be in a position to agree or disagree? Mr Henderson: Well, he might be if he recalled. Mr Justice Fraser: Do you recall that happening when there were outages? A. No. I cannot recall. Mr Justice Fraser: Were you aware of when outages would occur like that? A. Not all the time. Mr Justice Fraser: Do you want to put the question again? Mr Henderson: Yes. What I'm suggesting is that the cause of this problem was that an outage occurred at a particular point in time. A. Yes. Q. You were in the process of effecting a transaction from POCA? A. Yes. Q. It was in the stack and it had cleared from POCA? A. Actually you mentioned the word "outage". I'm not -- was there a power outage? Q. I'm not sure if it was a power outage, but I think it may have been a problem with the system. A. Ah, okay. Q. The system went down in some way. A. So -- yes, because you said that if there's a power outage then there's evidence when you have to log back in, so did that happen on this occasion? Q. Okay, I want to come to all this and I'm doing this clumsily. What I'm suggesting is that what may have happened -- and if you don't recall, you don't recall, but what may have happened is that the transaction was in the stack, the money had been taken from the Post Office card account and before you cleared the stack there was an outage . A. Possibly. Q. Okay.”
“Date:14-Dec-2011 10:43:53 User: Wayne Bragg Summary: The banking transaction had completed (A3 rec'd and authorised @ 13:33:37), including the receipt print (13:33:42), and money should have changed hands. The basket settlement failed from 13:35 with 'No response received from data centre', and the two retries also failed, and the attempt CANCELLED at 13:37. The Disconnected Session receipts show "Cash TO CUSTOMER 195.04" so the customer's account should be correct but the branch will have a shortage (for a withdrawal) because the session hasn't been recorded.”
“Your enquiry has been investigated and I can confirm there are no issues with Horizon online or the production of postage labels which would cause the situation you described. However if a user either pressed the yes or return key quickly before screen messages appear this can lead to a user confirming a postage print has happened when in rare circumstances it may have failed and put the cost of the failed print into the basket. In this circumstance Subpostmasters should contact Network Business Support Centre to arrange a credit for the spoiled postage.”
“ Q: There were procedures built into Horizon to cater for the situation that you explained -- I have to say in the vaguest of terms, but as I understand what you are saying, there were procedures in place which ensured you could deal with the situation, weren't there? A. No. Q. We will have to differ.”
“Q. Well, what I'm suggesting is that the most natural explanation for this, whatever adjustments were made, is that at some point on 11 May someone hadn't counted a big pile of£10 notes. They had been put in a safe and forgotten about, which is understandable, and that they were found the next day, or located, and there was an accurate cash declaration on 12 May. A. I don't know about that, sir. Q. It's perfectly plausible, isn't it? A. I can't say anything to that.” 124. This suggestion by the Post Office is, in my judgment, somewhat fanciful. A “big pile of£10 notes” is not the sort of thing that ordinarily gets “put in a safe and forgotten about”, and then suddenly “found”
“Q. I suggest finally, Mr Patny, it is rather surprising, given that you do all the balancing, or did all the balancing in the branch, that you don't understand that the effect of increasing the declaration of stamps is to have a corresponding effect on the declaration of cash? A. I wasn't aware of that, no.” 134. There was no direct evidence to support the proposition that by increasing a declaration of stamps, it would have a corresponding effect in the opposite direction upon the declaration of cash held in the branch. In any case, in the Common Issues trial, there was evidence that making a cash declaration required specifying how many notes of each denomination was physically held in the branch. An SPM had to declare both the cash and the stock that was held in the branch. It is difficult to see how changing the number of stamps held in a branch would, or could, affect the physical counting and declaration of the amount of cash held in a branch by way of specific denominations. However, and in any event, the allegation of dishonesty against Mr Patny was a line of cross-examination that was not supported by evidence from Ms Van Den Bogerd, who was expressly asked about it, and who accepted that she had not made an allegation of dishonesty in her witness statement and said she did not offer an opinion on whether Mr Patny was dishonest or not. That means that there was no evidence from any witness called for the Post Office to support the allegation of dishonesty. 135. The second problem in relation to which Mr Patny gave evidence was a MoneyGram transaction in February 2016. This related to a failed payment by card by a customer, who tried to send£3,100 by MoneyGram but whose card was declined twice. However, the branch accounts still showed two debits in that amount in the branch accounts that led to a loss showing at the branch of£6,200 . A TC was issued but only for one of these, which led to a£3,100 loss. Mr Patny accepted that he had only cancelled the transaction at the time, and not reversed it as well (which was required) until later, in the evening, when advised to do so by the helpline. As he put it, the amount had “doubled up”, in that the MoneyGram transaction, which was cancelled and reversed later on (which was for£3,100 ) had led to a loss showing of£6,200 . If that is correct, then it is indeed double the value of the transaction. Ms Van Den Bogerd said that this was “not supported by the data” and also relied upon the fact that the cash shortfall on the day was greater than£6,200 , namely about£500 greater than the cash shortfall which Mr Patny said had been caused by MoneyGram. She could not proffer an alternative explanation, however, and in her witness statement said: “On 23 February, the branch declared cash holdings of£25,803.87 , a net downward movement of£8,601.59 . The net value of transactions during this period resulted in a£1,806.71 decrease in cash. The Moneygram transaction described above would account for a further£3,100 , bringing the total explainable cash movement to a£4,906.71 decrease. However, this leaves£3,694.88 of cash movement unaccounted for. I cannot say for certain what caused this additional loss of cash but there is nothing in the accounts that suggest a problem with Horizon. It appears more likely to me to be a problem with cash handling in the branch or a user error when making cash declarations.”
“Following Mrs Burke's investigation, Post Office generated a transaction correction for the£150 withdrawal that had not been recovered at the time. If Mrs Burke had not done this herself, there is a process built into Horizon for flagging non-recovered transactions which would have prompted an investigation and I'm sure would have led to the same outcome.” 151. She explains that had this not been done, more would have been required from the Post Office – namely an investigation – before the Post Office would have reached the decision to refund the£150 , although she does say that she is sure the same outcome would have been reached. This is rather different to the way the case was put to Mrs Burke, which did not include the requirement for any such further investigation by the Post Office being required. The impression sought to be given was that on the Fujitsu documents already available, a TC would have been issued anyway to Mrs Burke a few days later. I do not accept that. Something more at the Post Office end would have been plainly required, as Ms Van Den Bogerd explains. Mrs Burke was the person who had in fact provided proof of payment out, and so no investigation was undertaken. 152. It is in my judgment likely, on the basis of all the evidence in the litigation to date, that the sum of£150 would have showed up as a shortfall at the end of the next branch trading period, as any further investigation was not likely to have been completed by the end of that period. However, if Mrs Burke had not acted as she had, and if an investigation had been done by the Post Office, if that had been resolved in her favour, and if that had led to a TC being issued, then the shortfall in the branch accounts would have been corrected in a later branch trading period, namely the period during which the TC was issued. No loss flowed to Mrs Burke in respect of this incident, as a result of her own investigation. In any event, this is not a point of difference between the parties that it is necessary to resolve in order to resolve the Horizon Issues. There was plainly a potential impact to her branch accounts in any event, and that potential impact was caused by the Horizon system. Mr Roll 153. Mr Richard Roll had worked at Fujitsu between 2001 and 2004. This was during the days of Legacy Horizon and he had no experience of Horizon Online. He provided two witness statements, the first of which was dated11 July 2016 . This was obviously very much in the early days of the group litigation. The second statement addressed certain factual matters in Dr Worden’s first expert report. I consider Mr Roll to be an important factual witness in this group litigation. He has no personal interest in the litigation and is not a claimant. He has never worked for the Post Office, although whilst at Fujitsu working on Legacy Horizon he obviously had a great amount of involvement in the Horizon system. However, given the expert evidence and particularly the degree of agreement between Mr Coyne and Dr Worden on the number of accepted bugs in Horizon, which even on Dr Worden’s position was 11 different ones, in addition to Mr Roll’s evidence there was ample other evidence in relation to my findings on the Horizon Issues. Mr Roll’s career has started in the Royal Air Force, which he joined in 1976 and left in 1989. Whilst in the RAF, which he had joined as an avionics engineer, he worked on mainframe computer systems and was selected for a software development team working on aircraft control and attack systems. After he left the RAF, he worked in various roles in development and support, and joined Fujitsu in January 2001. There, he worked in the Software Support Centre, or SSC, in Fujitsu at Bracknell in 3 rd line support. 154. After he left Fujitsu he changed career direction entirely and attended the University of Southampton, from where he obtained a BSc in Podiatry. He worked in the NHS until 2011 and then went into private practice, where he remains, now with his own clinic. The claimants described Mr Roll as a “whistleblower”, effectively making public allegations about how Fujitsu dealt with Legacy Horizon and the access which was available to Fujitsu to SPMs’ branch accounts. This was access of a far wider nature than admitted to by the Post Office or Fujitsu publicly. 155. It was Mr Roll’s evidence that was substantially behind the eventual acceptance by Fujitsu witnesses of the ability to obtain remote access without SPM knowledge or permission, and the injection of messages into the counter. In particular Mr Parker had, in his first statement, described that Fujitsu lacked this “power” and had said that Mr Roll’s evidence was inaccurate and misleading. Mr Parker later corrected his 1 st statement, and I deal with his explanation about this change – and what he said he meant in his 1 st statement – in my summary of Mr Parker’s evidence below at [464] to [498]. 156. Mr Roll described himself as an IT specialist, a description with which I agree, and one which is demonstrably correct given the number of years he performed his role at Fujitsu in the SSC in 3 rd line support. Problems or issues would only reach 3 rd line support if 1 st or 2 nd line support did not resolve them. His IT experience includes, in my judgment, a considerable degree of expertise in software. Indeed, although I am critical of Fujitsu elsewhere in this judgment, that criticism does not extend to any suggestion that they employed personnel within SSC who were not experienced and sufficiently expert in software matters to perform that role. Had Mr Roll not been sufficiently qualified to work in 3 rd line support, he would not have been there for the period that he was. 157. The Post Office in cross-examination attacked both his recollection, and seemed also to challenge his expertise. This the Post Office is entitled to do. Given he was 3 rd line support, in my judgment he obviously had a high degree of expertise. There was a debate at one stage about whether he was “elite” or “super elite”, and he put himself in the former category. The Post Office also concentrated on a narrower band, or sub-set, of the issues regarding which he gave evidence, focussing more on potential problems with code and Mr Roll’s experience of those, rather than the wider issues generally in respect of Horizon of which he gave evidence. He had made clear very early in his cross-examination that, so far as he was concerned, data corruption was an issue in Legacy Horizon as well as software issues which were related to the code. 158. Mr Roll had left Fujitsu in 2004 and there will inevitably be difficulties in recollection by any witness of detailed points after such a passage of time. He was willing to indicate when he could not remember something, and would agree with points put to him by the Post Office when relevant. These were often on the basis of hypotheses which he was asked to accept, often in relation to a spreadsheet which Mr Parker had prepared showing how relatively few software issues Mr Roll was said to have worked upon. Mr Roll also, again inevitably, was not submerged in the fine details of the subject matter of the litigation, in the same way as some of the Fujitsu witnesses such as, for example, Mr Parker himself, who is the Head of Post Office Application Support and the SSC, and who has worked on the Post Office account at Fujitsu since 1997, before Horizon was even introduced. Given Mr Roll left Fujitsu in 2004, it is 15 years since he worked on Horizon. 159. His two statements were fairly short (4 and 8 pages respectively) and given at a high level. One example can be provided from his cross-examination. He was of the view that problems referred to 3 rd line support, which by definition would mean that 1 st and 2 nd line support had not been able to deal with them, would not include problems caused by mistakes by SPMs. Mr Parker disagreed with this and this was put to Mr Roll. “Q: Then the next unnumbered paragraph, Mr Parker says: "If NBSC were unable to identify the cause of a discrepancy they would often fall back on a default statement along the lines of 'this looks like a software issue' so that the SSC would investigate it. However, Mr Roll's statement that 'if an error was referred to us then it was extremely unlikely to be due to a mistake made by a postmaster' is not correct. The vast majority of discrepancies investigated by the SSC as pseudo 'software issues' were (and are) not caused by software issues." I would like to suggest to you, Mr Roll, that that's true, isn't it? A. The way that I remember it, it was issues to do with the software that were causing the problems. Whether that was the programme that had been written or data corruption, that's what I remember as our problems being. Q. You remember there being problems with data corruption? A. Yes. Q. I'm not asking you about data corruption, Mr Roll, I'm asking you about software issues. The claim I would like to put to you again is the last sentence of that paragraph: "The vast majority of discrepancies investigated by the SSC as pseudo 'software issues' were (and are) not caused by software issues." Are you in a position to agree to that? A. From my recollection I would disagree with that, but it was a long time ago. Q. I'm grateful.” 160. This passage of cross-examination, and there are a number of other similar examples, shows the following: 1. Mr Roll’s conclusion was a general one, with which Mr Parker disagreed. Assertion and counter-assertion are not, of themselves, of assistance to resolving the Horizon Issues. His conclusion, as with that of Mr Parker, is simply their point of view. 2. If a problem made it to 3 rd line support, then by definition it had not been capable of being resolved by 1 st and 2 nd line support. 3. Fujitsu would, even at the 3 rd line support level, attribute some problems which they could not understand or resolve to SPM error, and Mr Parker’s evidence is consistent with this approach by Fujitsu. This is made clear in the text of PEAKs put to Mr Parker. There is a very stark example in one of the PEAKs where the Post Office’s own auditors have specifically ruled out user error, yet Anne Chambers chose to close the PEAK with the defect cause being user error. 4. The Post Office would try to keep the evidence in cross-examination of Mr Roll within very narrow bounds. This passage shows that Mr Roll expressly gave data corruption as one of the problems of which he had experience. Horizon Issue 1 included the expression “bugs, errors and defects”
"Between1 January 2001 and31 December 2014 ...” and those are the years that you worked for Fujitsu "... the SSC received a total of 27,005 calls, meaning that on average 563 calls per month were dealt with over this 4-year period."
"Transferred calls (ie those not resolved by the SSC) are of interest."
"A very small proportion of calls transferred to 4th line support would have concerned software errors requiring resolution ..."
"... it would be interesting [therefore] to know the number of calls transferred to fourth line."
"... while the SSC have records of the volume of transferred calls, we do not retain records of where they are transferred to and it is not the case that all of these would have been transferred to 4th line support. For example incidents would often arrive at SSC from internal teams for routing back to help desks."
“A. When you take it as an average then yes I suppose so, but the system evolved from -- I started there in 2000 I think it was and left in August 2004. The nature of the work changed over time and the ones that stick in my mind are the ones where there was sort of the fire-fighting efforts where it was -- there were difficult periods where there were software issues, so ... Q. Yes, Mr Roll. We could by all means go back to the graphs because you will have seen from the live PEAKs into and out of the SSC it actually gave monthly figures, but what I'm suggesting to you, Mr Roll, is that -- although you may well have been busy on all sorts of things, the fact of the matter is that software problems requiring a software fix represented a tiny fraction of the work that was handled by the SSC third line support. A. Yes.” 166. This theme of the way that the Post Office put its case carried on into the evidence of Dr Worden and his Section 8 statistical exercise, which I address below. The expressions “tiny proportion”, “tiny fraction” and other terms used sought to demonstrate, basically, that Horizon worked very well most, if not all, of the time. Given the use to which the Horizon system was put by the Post Office, namely monthly balances which governed the branch accounts of SPMs, with shortfalls and discrepancies being “made good” – which means paid – by the SPMs in question, this approach can be seen to have limited utility. Firstly, these are subjective terms. Secondly, given the PEAKs show that a SPM may report a single unexplained discrepancy for (say)£25,000 , or even£1,000 or£500 , the fact that this may be only a tiny fraction of the number of calls the SSC worked on that month might come as cold comfort to that specific SPM. The number of calls, and how they were dealt with, is of some relevance to robustness, but such high level and subjective points were not likely to be of enormous assistance, one way or the other, in challenging Mr Roll’s evidence. 167. It was put to Mr Roll that his role, which he described as “product specialist”, was “the junior level of people working” in SSC, which he disputed. He explained that most people in SSC were “product specialists”. 25 of the 30 or so total number of people were, according to Mr Roll, all product specialists. “Q. -- the SSC? Of that 30 people how many people were at your level? A. When I worked there I -- it was two or three people were senior levels, I think Mr Parker was one, a couple of others, then there was Mike Peach and I believe the rest of us -- 25 or so. Q. The figures I would like to ask suggest to you, Mr Roll, are about 25 people had the junior level and there were about five people who were true specialists who I think you fairly described as specialist earlier on in your evidence. Would you accept that that might be the case? A. Junior level seems to -- it doesn't fully explain the complexity of the system or the knowledge of the system required, but yes, I suppose ... Q. Well, let's agree on ordinary -- I'm not seeking to cast any imputations and it is right that you should – words do have implications. So the ordinary level was your level – A. Base level. Q. -- and there was this perhaps five or so people above -- the senior people above you and what did they do? Did they do the more challenging work?”
“If we were unable to find the cause of the discrepancy then this was reported up the chain and it was assumed that the postmaster was to blame.” 174. He said that even if software fixes were developed the problem would sometimes reappear several weeks later. He also stated that remote access to branch systems was possible; the ability was extensive; that this was done without the SPMs being aware; that data and transaction information could be changed by Fujitsu; and that sometimes SSC would log into a branch system whilst it was switched on but not in use. 175. The final paragraph of his 1 st statement said: “In summary, the issues with coding in the Horizon system were extensive. Furthermore, the coding issues impacted on transaction data and caused financial discrepancies on the Horizon system at Branch level. It was those issues that I, and other colleagues at Fujitsu, were routinely working on daily. Furthermore, remote access to the Horizon system at Branch level was extensive, as was the ability to change data and change transaction information, even while the postmaster was working, without the postmaster being aware of this.” 176. He had also said that “during the course of resolving the software issues, we would frequently access a Post Office counter IT system remotely”. 177. His use of “frequently” and “routine” are, in my judgment, subjective, and as explained above in terms of “tiny fraction”, subjective terms are not entirely helpful. What he meant by this is it was not unusual for this to occur. It is difficult to judge, at the remove of 15 years from when Mr Roll left Fujitsu, just how often something that he remembers as frequently or routine in fact occurred. He sensibly accepted that the more regular or mundane matters in which one is involved tend not to be remembered, with rather more unusual events sticking in the memory. In terms therefore of how often this occurred whilst he was there, it is not possible to come to a concluded view. However, I consider it important that the occasions to which Mr Roll refers were not isolated incidents on his evidence, nor were they unusual, and that financial discrepancies on the Horizon system at branch level was something which he recalls working on daily. It is also important that he gave evidence that if Fujitsu could not track down the cause then it was assumed the SPM was responsible, in other words user error would be used as a default setting for investigations. This matches the evidence of an enormous number of PEAKs in the Technical Appendix. 178. He also gave evidence about hardware failures, and gave specific responsive evidence in respect of Dr Worden’s expert report. 179. I accept Mr Roll’s evidence, which is supported by the contemporaneous documents. The Post Office in its closing submissions accepted that he was a careful and precise witness, and also that “he was at pains to assist the court and to give accurate answers”
“15. During my time at Fujitsu I know that there were budget pressures and redundancies which impacted system development and testing. The test team felt they were under enormous pressure to complete the testing within certain timescales which negatively affected the test regime. Meanwhile, the development team had to balance time spent on fixes, with time spent on developing new features for Legacy Horizon and time spent developing a new system which I believe later became Horizon Online. 16. In my first statement, I refer to the pressure that the SSC team and Fujitsu were under generally due to an awareness of the financial penalties imposed by the service level agreements between Post Office and Fujitsu (paragraph 12 of my first statement). I believe that although individual penalties were quite modest, when applied across multiple counters/post offices the cumulative figures involved were very high, potentially amounting to tens of millions or more. I disagree with Stephen Parker’s statement that these potential financial penalties were not a factor for the SSC (paragraph 43 of Stephen Parker’s witness statement) as we were aware of them and often commented on them, e.g. “That’s saved Fujitsu another£25 million ”.” 182. I do not attach much weight to this as pressure in the workplace is such a subjective matter. Some members of SSC may have felt under great pressure; others less so. This is a subjective account of what it was to work at SSC at that time from the point of view of Mr Roll. The type of statement to which Mr Roll refers in terms of millions of pounds saved does not advance matters one way or the other. Given the type of problems that 3 rd level support was involved in attempting to resolve, there would have been an inevitable pressure involved in attempting to resolve such issues speedily. Mr Roll chose to leave Fujitsu, and therefore he left those pressures behind. Putting it at its mildest, it would be extraordinarily disappointing if PEAKs were closed, attributing fault to a SPM, simply because that was the easiest and quickest way for a SSC product specialist to keep on top of their workload. Without separate cross-examination of each product specialist who chose to close PEAKs where prior entries suggested user error had been ruled out, it is not possible to determine the degree to which this was their motivation in any individual case. However, it is not relevant. It is no part of the Horizon Issues to determine why, or whether, individual personnel at SSC acted in a particular way – budgetary concerns for their employer; exercises of judgement; even laziness; or any of the other many possibilities. Mr Parker gave evidence that “the possibility of financial penalties was never a factor for the SSC”, and Mr Roll said that it was. Mr Parker’s evidence on this suffers from a wholesale attribution of this lack of motive to every member of SSC, as indeed does Mr Roll’s, in the other direction. Everyone is different, and there will have been a range of different reasons operating on each member of SSC each day. Any sensible business will, in any event, always have at least part of its attention on financial performance and this is understandable. The Horizon Issues trial is not an inquiry into how Fujitsu manages its personnel, or its business. I do question the allocation of Category A, B or C to some of the PEAKs that were used in the trial, but this is something that was pursued (though not very far) with Mr Parker and I deal with it there. 183. The way that Dr Worden was simply not prepared to accept Mr Roll’s evidence at face value, and set out to disprove his factual evidence, something which in my judgment an independent expert should never do, is dealt with in the section of this judgment where I deal with Dr Worden’s evidence. The degree to which the evidence of Mr Roll, and his experience of Legacy Horizon between 2001 and 2004, affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions. 184. The fact that I find Mr Roll reliable does not mean that I automatically accept all his conclusions. I have given a specific example at [181] about financial pressures. However, Mr Roll’s evidence was supportive of the claimants’ case and it was necessary for the Post Office to render his evidence unreliable, insofar as they could, in order to damage the claimants’ case and bolster their own. In my judgment, this the Post Office failed to do. Mr Roll’s evidence was also what directly led to the true picture emerging of remote access. However, of itself this does not provide an entire answer to the Horizon Issues. Mr Henderson 185. As explained in [78] and [79] above, Mr Henderson is a director of a company called Second Sight which was involved in a mediation scheme, until the Post Office withdrew from it. Initially Second Sight was engaged to perform an investigation, but after its interim report its scope of involvement changed. It ran from 2012 to 2015, when Second Sight’s appointment was terminated. The investigation and subsequent mediation scheme were set up in the light of disquiet about the complaints of various SPMs, the operation of Horizon, and concerns that were publicly discussed, not least by Members of Parliament. All of the documents and emails which Second Sight had at that time were returned to the Post Office. Mr Henderson estimates these as 16,500 emails and 18,500 other documents, which he said were sensibly organised in 1,700 folders. They were in good order. 186. The documents in that set that were provided by the Post Office to the claimants for this litigation were not, according to Mr Henderson, provided in the same way. He criticised them for having no folder structure, having had metadata removed, and lacking the original date and time stamps which were present on the original documents with which Second Sight had been provided by the Post Office. Some important documents, such as the Fujitsu XML transaction reports, were described by the Post Office as “unreadable”
“2.1 The Scheme has been set up to try to achieve the mutual and final resolution of a Subpostmaster's concerns about Horizon and any associated issues. 2.2. Second Sight Support Services Limited ("Second Sight" or "you") has agreed to be a member of the "Working Group" whose role it is to oversee the Scheme and to assist in investigating individual Subpostmaster complaints. 2.3. This letter and its schedules will form the basis of the terms of Second Sight's engagement by Post Office Limited ("Post Office") to provide Services to the Working Group in relation to the Scheme.” 191. Clause 6.2 stated: “6.2 Second Sight will not, and will ensure that the SS Directors and any SS Personnel will not, act, directly or indirectly, in any capacity (whether for any former or current Subpostmaster or a competitor of Post Office or otherwise) against Post Office or any of its officers, directors or employees save to the extent a) that it is expressly agreed in writing by Post Office that the work proposed to be undertaken will not have a material adverse effect on Post Office's commercial or financial interests or reputation, or b) as required by applicable law or by the mandatory rules or requirements of any regulatory authority, government department or agency to which Second Sight is subject or c) as required by an order of a court of competent jurisdiction.”
“I will not act, directly or indirectly, in any capacity (whether for any former or current Subpostmaster or a competitor of Post Office or otherwise) against Post Office or any of its officers, directors or employees save to the extent a) that it is expressly agreed, in writing by Post Office that the work proposed to be undertaken will not have a material adverse effect on Post Office's commercial or financial interests or reputation, or b) as required by applicable law or by the mandatory rules or requirements of any regulatory authority, government department or agency to which Second Sight is subject or c) as required by an order of a court of competent jurisdiction.”
“Mr Henderson was asked whether confidentiality restrictions had caused him any inhibition in answering the questions put to him in cross-examination. He said that he had the issue in the back of his mind and that he had tried to make sure that his answers did not infringe the protocol. It is understandable that Mr Henderson would wish to be careful, but the idea that he was restricted in answering Post Office’s questions is bizarre.” 196. The submissions then went on to deal with the questions put to Mr Henderson, and how there was no restriction that could have impacted upon his answers. 197. The actual question posed to Mr Henderson was not as set out in the above extract of the submissions at [195] above, it was rather wider. It was posed by me and it was at the end of his cross-examination. It was in the following terms: “I just want to be clear: is it your evidence therefore that because of that protocol agreement your evidence of fact to this court is narrower in scope than it would be absent the protocol agreement?” 198. His answer was “yes, it is”
“Q:…. If we could just look at it very briefly, the termination letter to which you have just referred is at {F/13/24.1}. That's the termination letter I think to which you at least indirectly referred. A. That's correct. Q. If we could look then to {F/1324.2}. This is a much longer letter from Post Office that came to you on the same day. Do you recall this longer letter? If I maybe tell you broadly what it is to do with. It's a letter in which Post Office sets out a plan for how Second Sight could finish its outstanding work. Do you remember that letter? A. Yes, I do. Q. And the second paragraph makes clear that Second Sight was expected to continue working during the notice period and that even beyond the notice period there would be a proposed future role for Second Sight. That's what this document dealt with, isn't it? It's a fairly long document. A. It is, but there's another document of that date or very close to that date which you haven't mentioned which was the press release from Post Office announcing the winding up with immediate effect of the mediation scheme itself. That I understand was the primary decision and our termination was a consequence of that, not a separate issue. Q. Well, I think when you say mediation scheme it's fair to say, isn't it, that this winding up process, as you describe it, did not put an end to mediations; in fact – A. It put an end to the mediation working group with, as I understand it, no consultation and the announcement was made I think the day immediately before the next planned meeting of the working group, so it was a considerable sort of shock to everybody.”
“The court will generally uphold that confidentiality [of mediation proceedings] but where it is necessary in the interests of justice for evidence to be given of confidential matters, the Courts will order or permit that evidence to be given or produced.”
"Phantom transactions have not been proven in circumstances which preclude user error. In all cases where these have occurred a user error related cause can be attributed to the phenomenon"
"Given the current media and in particular the BBC's attention on Horizon, do you think it is worthwhile looking into this 'alleged flaw' with Horizon that this SPMR has highlighted to pre-empt any enquiries from his MP?" 218. The ultimate response from Andrew Winn of the Post Office was that the claim could not be investigated without further details and Fujitsu involvement, that Mr Winn did not understand the purpose of the call by the SPM, and also stated: “My instinct is that we have enough on with people asking us to look at things.” 219. Mrs Van Den Bogerd agreed that this was an inadequate response. She said it would have been very easy for Mr Winn to have contacted the branch and obtained further details, and he should have done so. In my judgment, the stance taken by the Post Office at the time in 2013 demonstrates the most dreadful complacency, and total lack of interest in investigating these serious issues, bordering on fearfulness of what might be found if they were properly investigated. This SPM, whose branch was known to the Post Office, should obviously have been asked for further details (if further details were required for an investigation), and the Post Office and/or Fujitsu should plainly have investigated the matter as a matter of some importance. By 2013 Horizon was an extraordinarily controversial subject; there can simply be no sensible excuse for the Post Office’s failure to try and understand this particular subject. This is particularly reprehensible given that an internal Post Office document in August 2013 showed that Mr Winn’s involvement in this was because his area of responsibility was as follows: “also responsible for resolving specific branch accounting issues.” It was his specific job to resolve specific branch accounting issues, yet he decided at the time that “we have enough on”. I agree with Mrs Van Den Bogerd that this is inadequate – that is putting it at its most favourable for the Post Office. Somewhat stronger terms are also justified. 220. Mrs Van Den Bogerd had only learned of the ability of Fujitsu in terms of remote access, namely the insertion of transactions at the counter under Legacy Horizon “within the last year or so”, which given she is central to the Post Office’s position in the group litigation shows, in my judgment, a remarkable situation. It is not necessary, in order to resolve the Horizon Issues, to go further into why she has found this out so belatedly, or why that might be. 221. Mrs Van Den Bogerd would accept changes to her evidence in cross-examination where these were justified. One example was in relation to Mr Latif and corrections issued in respect of the lottery. Her written statement had said: “However, due to an error by Post Office, instead of increasing the scratch card stock, the TAs decreased the stock. To be clear, this was a data entry error by Post Office and not an issue with Horizon.” Actually, given the process of issuing TAs for the lottery was entirely automated, and done in conjunction with the Camelot system and by Horizon, she accepted that the passage should be corrected. The cross-examination was as follows: “Q: So would it be fair to correct that part of your statement – A. It would be actually, yes. Q. -- to say "
“Q: The extracts taken from the report by Helen Rose referred to by Mr Coyne are taken out of context and mistakenly claim that the relevant reversal was issued in error by Horizon not the subpostmaster." I just want to give you an opportunity -- Mr Green was pressing you to accept that actually what you had said in that sentence was wrong. I would like to give you an opportunity to explain to the court what you actually meant. A. What I meant was that the actual reversal was part of that recovery and it had actually taken place as it should have taken place, which is what I meant in that. So it wasn't a failed reversal because it actually had happened as it should have happened, but I accepted in there that the -- it wasn't obvious to the postmaster at the time that what had happened -- that he hadn't -- because it didn't show that he had actually -- it showed that he had done it and he knew he hadn't done what we referred to earlier was an explicit reversal. That's what I meant in that.” 226. I do not accept Mrs Van Den Bogerd’s characterisation of the evidence she had originally given in her witness statement. She had clearly stated that Mr Coyne, having taken passages out of context, “mistakenly claim[ed] that the relevant reversal was issued in error by Horizon, not the Subpostmaster”
"Q: “Just one question from my part - if the reversal is system created but shows as an existing reversal, could this not be reflected with a different code ie SR (system reversed) to clear up any initial challenges. My feelings at the moment are not questioning what Horizon does as I fully believe that it is working as it should, it is just that I don't think that some of the system based correction and adjustment transactions are clear to us on either Credence or ARQ logs."
"I understand your concerns. It would be relatively simple to add an extra column into the existing ARQ report spreadsheet, that would make it clear whether the reversal basket was generated by recovery or not. I think this would address your concern. I'm not sure what the formal process is for changing the report layout. Penny, can you advise as to the process: is this done through a CR?"
" I do believe that the system has behaved as it should and I do not see this scenario occurring regularly and creating large losses. However, my concerns are that we cannot clearly see what has happened on the data available to us and this in itself may be misinterpreted when giving evidence and using the same data for prosecutions. My recommendation is that a change request is submitted so that all system created reversals are clearly identifiable on both Fujitsu and Credence."
"The system is behaving as it should."
“Let's take it in stages. You can see there it says "The system is behaving as it should"? A. Yes. Q. That would be problematic for a subpostmaster, wouldn't it? A. The way Gareth [Jenkins] describes it here would be, yes, because what he says is that it would have printed the session receipt but it doesn't seem as if it did, when actually disconnection transaction receipts were actually printed in this example and a recovery receipt was printed. But that's not referred to in here. Q. Well, let's just have a look at that. Let me just ask you my second question and then we will go on to probe that with more care. The second point is are you aware of a closure code for Fujitsu for PEAKs which refers to faults which are known in the Horizon system but agreed between Post Office and Fujitsu to stay there? A. I'm not aware of a closure code. Q. You didn't know about that? A. No, I don't know about that.” 229. The following important points arise in respect of this: 1. Mr Jenkins thought that “the system had behaved as it should”
"For the last several months Post Office has experienced a live operational issue with MoneyGram transactions across the branch network . In the event of a transaction timing out at the counter, a system error message is displayed to the user ... and the transaction is aborted. This leaves no record of the transaction at the counter and the transactions and funds may or may not have been committed in the MoneyGram domain. This causes significant issues for Post Office and MoneyGram and for customers."
" Failure to be open and honest when issues arise eg roll out of Horizon, HNGx migration issues/issues affecting few branches not seemingly publicised."
“PM - Dugannon PO£43k discrepancy Outlet has a discrepancy of£43,000 after balancing SUs and doing office snapshot. Phil Turnock POCL BSM has advised outlet on this week’s balance. Steve Warwick development is investigating why this mis-balance occurred. Immediate impact of this week’s balance has been addressed but POCL are concerned that the cause is still unknown and this will affect this and other outlets.” 237. After some investigation the following entry appears: "I have talked with development ref this problem. It is seen as a one-off. No fault can be found and developments do not expect to be able to find a fault with the evidence available. There is no additional information available as evidence. I suggest this call be placed on monitor for 1 month." The approach to this discrepancy mirrors so much of the case. A discrepancy occurs; Fujitsu cannot find a fault; and say they do not expect to be able be find one “with the evidence available.”
“Further examination of the event logs for these two counters indicate that counter 5 looks suspect (C drive nearly full and big gap of no messages). Calls from PO into HSH for period between 30-Oct and 10-Nov indicate a reboot (counter not specified, but would tie in with counter 5 event log) on Saturday 30-Oct-1999. The evidence in the message store was that messages continued to be written to the message store but that all the 'Payment' transactions which should have been recorded in the rollover trailer messages failed to appear (although others did, such as the Rem OUT and Transfer OUT totals). This indicates that the problem was not one of running out of Disk space but of failing either to retrieve, or write out, transaction totals for one particular node in the node hierarchy. Given that there were known problems with corrupted Persistent Object indexes at about this time, it is possible that an update to an EPOSSNodes object failed to be registered correctly at the outlet, causing the node accumulation to fail. It was decided to prove this out by deleting the 'Payments' node in the node Hierarchy and then running the SU balance, to attempt to identify the root cause of the problem. Call passed to testing to be scheduled. Update18th Feb 2000 The test was carried out on 16th February as follows: delete the Payments EPOSSNodes object before producing a SU balance, on a version of the current live system (CI2_2R). When trying to print the Payments part of the SU balance, the missing node is detected by the system and an error tablet with message "A system error has occurred whilst printing. Please ring the helpdesk. Error at 67640." is generated. So the balance could not be finished. This type of error trapping error trapping was introduced at the end of last year when resolving AI298 issues and we are investigating if the outlet did not have such error handling when the problem occurred. Certainly, with the current system, a missing Payments node now would not go undetected. The problem is currently back with development for further investigation.” 238. On17 March 2000 there is another entry that says there has been another incident “of a very similar nature” at another office. This can therefore be seen to be something that was not, as recorded initially, a “one-off”, even if that were acceptable given the discrepancy of£43,000 . Mrs Van Den Bogerd agreed that this showed that the underlying information was not checked until February 2000, even though the discrepancy occurred in November 1999. She was asked about a later entry in April 2000 that referred to another occurrence, this time in Appleby for£9,000 approximately. The cause for this was recorded in July 2000 as “data trees failing to build” and that entry states “there have been a number of calls relating to this kind of issue. A fix has been put in at C14 which will prevent this happening”
“On week 9 receipts and payments misbalance of£1337.05 , week 10 misbalance of£24000 , week 11 misbalance of£12000 , week 12 misbalance of£1051111.48 and week 13 misbalance of£17426.05 , she has a difference on week 11 of balance due to post office and balance brought fwd on week 12 of£1082544.32 Overall these weeks net out a difference of£27343.84 . she needs business support (reconciliation) to look into this.” 240. Mrs Van Den Bogerd agreed that£1.05 million was a large amount for any branch and could not be a “real amount of cash” that “the SPM has put in her pocket”
“Balance brought forward was multiplied twice due the known software error. The initial balance brought forward for this CAP was£1196622.72 . This was multiplied twice to give a total BBF of£2279189.04 . The discrepancy was therefore£1082540.28 . This was due a known software error which has no been resolved.” 241. She did not know if “no resolved” meant “now resolved” or “not resolved”
“I have now taken over analysing this problem from Steve Warwick who requested additional information on13/10/1999 17:49:21. I have looked at the information and I am not convinced it can possibly be complete. For example he said he wanted information from 20th May until 26th May. I opened the spreadsheet mc20may.xls and observed it only had 6 counter 32 transactions in - is this correct? In any case my team is not used to dealing with data in this form (spreadsheets). We have developed techniques to look at problems using a full message store plus audit and event logs from the failing counter. Even if all the relevant spreadsheets could be obtained I do not think it would be worthwhile me getting to grips with this new method of analysing problems. I see this is a very old problem (21/07/1999 ) and there have been many software updates since then. May I suggest we discontinue investigation of this particular problem but that if a similar problem occurs again you send full message store plus audit and event logs from the failing counter.”
“The most frequently occurring incidents in November were both types of Receipts and Payments Incidents (Migration and Post Migration), with 31 incidents per category. The Migration incidents have remained at the same level, whereby the Post Migration occurrences have increased. This was followed by 17 Transactions Polled by TIP but not by HAPS, these were due to delayed transactions as reported on APSS 2133c. These transactions are added back into normal processing.” 243. Mrs Van Den Bogerd was not aware of these problems going on at the time; she was a Regional Network Manager and did not have a national role then. No background information was provided to her then about problems that branches other than her own were experiencing, which were being passed back to ICL at the time. She therefore had no knowledge of the type of problem experienced by Mr Bates in June 2001, which was recorded in a Post Office audit document of his branch that was put to her which stated “A correct assessment of cash holdings could not be made because the Horizon system intermittently adds the previous days cash holdings to the daily declaration.” 244. By 2010 and the introduction of Horizon Online, she did have a national role and broader responsibility. She agreed that Horizon Online did not upgrade functionality, and documents were put to her that showed, as Mr Green put it “a litany of issues” with Horizon. The pilot scheme went sufficiently badly that Fujitsu put it on “red alert” and the high volume pilot was suspended in April 2010. Mrs Van Den Bogerd had not even mentioned this in her witness statements, and said she did not know this had occurred. The NFSP are recorded as “raised concerns but remain supportive”
“* Reduction of operating costs by£3m per annum * Reduction in net agent debt by£1m * Reduction in subpostmaster suspensions as a result of audit shortages to a level of 60 per year * Reduction of calls into NBSC by 25% * Reduction of audit losses of£10k of over by 50% * Satisfaction with on-line training models of 95%” 247. Apart from the first bullet point, three of the other five are consistent with an aim by the Post Office to make improvements to reduce the amount of debt incurred by, and suspensions, of SPMs, and reduce the audit losses. The other two are related to training. This is not consistent with a view that the debt/suspensions/audit losses are incurred by carelessness on the part of SPMs or criminal activity. It is also hard to see how it could be justified that these had been redacted originally. 248. So far as her evidence about Mr Latif was concerned, after Mrs Van Den Bogerd had been taken through the features of his experience with the failed stock transfer and the similarities with accepted bugs in Horizon, namely the printing out of two receipts very close to one another (in his case, 4 seconds apart), and asked if this worried her, she said she “would want to go back and have a look at” it. In my judgment, part of the way through the Horizon Issues trial in 2019 is rather too late for that. I conclude that there was no real answer available to Mrs Van Den Bogerd to the points put to her concerning Mr Latif’s experience. Had there been, I have no doubt she would have provided that answer in evidence. 249. Mrs Van Den Bogerd was in the witness box for in excess of one day, the longest period of any of the witnesses of fact for either the claimants or the Post Office. Her cross examination led to a far greater understanding of the Horizon Issues on the part of the court, although her written evidence was, as originally drafted, extraordinarily one-sided. She minimised any reference to problems or issues with Horizon, and reverted to potential user error whenever possible as a potential explanation, an approach which she explained in her written statement as providing “plausible” explanations. Her witness statement also stated, in terms, the exact opposite of what the reality of the situation was, and I have given examples at [221], [223] and [226] above. Witness statements are supposed to be factually accurate, and care must be taken in future rounds of this group litigation that they are drafted in accordance with the rules. Making statements that are the exact opposite of the facts is never helpful, to put it at its mildest. It is also the opposite of what witness statements are supposed to be. 250. However, during her cross-examination Mrs Van Den Bogerd readily conceded a number of points which were put to her. Her accuracy in her oral evidence was also more satisfactory than it had been during the Common Issues trial. She accepted her written evidence suggesting a causative user error by Mr Tank was not correct, for example. I do not consider that her written evidence had provided plausible explanations. It provided explanations that the Post Office wished to advance, as these explanations would, if accepted, provide a defence to the claimants’ factual case. These explanations were not based on the facts. 251. There were no evident attempts on this occasion to mislead me in her oral evidence, or avoid uncomfortable points through a claim of ignorance (with the possible exception of the PEAK referred to at [210] above) or of not having seen a document before. When she told me that she had not seen a particular document before, I accept that. However, she also told me that she had been assisted in preparing her evidence by a team of ten people, and the Post Office had devoted “high level resources” to her evidence, as one would expect. It is therefore very surprising that she had not seen many of the important documents that Mr Green put to her in cross-examination. Some, such as the master PEAK on phantom sales, one of the recurring issues on Horizon, would have been the obvious place for anyone, still less a team of ten, to start when considering preparation of evidence for a witness statement. She explained that there was some pressure of time in terms of how long was available to her to prepare the statement, and this was further explained in supplementary re-examination, but I do not accept insufficient time as a valid explanation for her lack of knowledge on such important points. For example, she told me that at the time of preparing her witness statement, she had not even heard of the Callendar Square bug, one of two bugs that the Post Office accepted some time ago had been present in the Horizon system. This is an extraordinary gap in her knowledge. She did not know that there was a KEL dealing with failed recoveries, originally raised by Anne Chambers in Fujitsu as long ago as 2010, which is called KEL acha959T and which was updated most recently in 2017. This described failed recoveries, and seemed on its face to accept that these would recur, and was very close to the experience of both Mr Tank and also Mrs Burke. I do not see how Mrs Van Den Bogerd (assisted by her team of ten, and with the benefit of the Post Office’s considerable resources) could seek to give accurate evidence in the Horizon Issues trial without referring to this KEL, still less without even knowing about it. I am also somewhat disappointed – putting it at its very best for the Post Office – that a team of ten could have assisted Mrs Van Den Bogerd in preparing a witness statement that was so inaccurate on such important points as I have identified above. 252. Although my findings on her evidence in the Common Issues trial cannot be ignored, I am of the view that her approach in the Horizon Issues trial to answering questions was far more constructive and aligned to what is expected of any witness giving evidence in court, particularly a senior witness of an organisation such as the Post Office. I do however consider that this litigation, and indeed her cross-examination, is a very expensive way for a senior director at the Post Office to become educated about the myriad issues contained in the documents that were put to her. Either the team of ten people assisting her with her evidence had the aim of producing entirely one-sided evidence in chief, or they were unaware of all the documents relied upon by the claimants. Either alternative is highly regrettable. 253. Finally, and this is a point in Mrs Van Den Bogerd’s favour, there is no doubt that the Post Office is not as sufficiently close to the detail of what has occurred over the years on Legacy Horizon and Horizon Online, as Fujitsu. As will be seen from my analysis of the Fujitsu evidence of fact, I have certain views about the lack of accuracy on the part of Fujitsu witnesses in their evidence. If that lack of accuracy has also been included in reporting to the Post Office by Fujitsu, then that goes some way to explaining the Post Office’s lack of grasp of so much material that is consistent with the claimants’ case. As at late 2019, the date of this judgment, the Post Office also has the added benefit of the views of both the IT experts in the litigation, their four joint statements, the agreed number of bugs in the bug table (12, if one takes the number accepted by Dr Worden), the total number of bugs Mr Coyne says he has discovered (which so far as Horizon Issue 1 is concerned became 21), and collectively my findings on the Horizon Issues. These are not restricted solely to the number of bugs which I have found to exist which relate to Horizon Issue 1, but also those that relate to Horizon Issue 4. There are multiple bugs, errors and defects in both Legacy Horizon and Horizon Online in its HNG-X form. 254. The degree to which the evidence of Mrs Van Den Bogerd sits with my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions. Ms Phillips 255. Ms Phillips is a Team Leader at the Post Office for Agent Accounting and Santander Banking. She was originally engaged by the Post Office in 1999 through an employment agency, and has worked in a number of roles since then. In her current role, she oversees the processes of recovering losses from SPMs that they have declared in their branches. The department used to have “debt” rather than “accounting” in the name, but its function remained the same. 256. Ms Philips confirmed what had been in dispute for so long during the Common Issues trial, namely that SPMs had no option but to accept the figures provided to them, even though they may have been “settled centrally”
“TCs are issued by FSC. I understand from Post Office’s solicitors that the processes by which FSC determines whether a TC is required is outside the scope of the Horizon Issues trial.” 270. The reason that this is important is as follows. Dr Worden relies upon TCs as one of his “countermeasures” that goes to the robustness of the Horizon system. Mr Coyne does not, and the claimants submit that the whole process of TCs is entirely outside the scope of the Horizon system. This evidence from one of the Post Office’s own witnesses makes it clear that the claimants’ point of view is shared (or at least was when Mr Smith’s statement was prepared) by the Post Office’s solicitors, and the decisions regarding whether TCs are issued are outside the scope of the Horizon Issues. This evidence by Mr Smith supports the claimants’ position on this, and also supports the approach of the expert evidence of Mr Coyne, and is contrary to the position adopted by Dr Worden. 271. A case management system was introduced by the Post Office in September 2018 to record each individual challenge to a TC. This is called the Dynamics System, and is still in what is called “roll out”
"There are a number of branch operations processes, especially around branch accounting and reconciliation, which operate using legacy processes. They are unnecessarily complex and detract Post Office branch resources from serving customers. Stock Unit Management and accountability is very poorly controlled and is operated on very complex business rules. The lack of accountability and visibility of cash and stock transfers between Stock Units can lead to errors, rework and provides opportunities for fraud ."
"Similarly, Suspense Accounting is based upon legacy cash accounting practices, ill-defined and out of date processes. Inefficiencies lead to poor utilisation of resources both in Post Office branches and Support Services." 280. Mr Johnson is a highly experienced person, as one would expect given that the Post Office chose him to train the legal teams and experts on Horizon. All his experience, so far as using Horizon is concerned, is in Crown Offices, which are not the same as branch Post Offices, in that the Post Office operates Crown Offices itself. This means that a SPM in a branch Post Office is personally responsible for any losses in that branch; whereas a Crown Office manager is an employee of the Post Office. This therefore gives him a somewhat different angle of approach to the use of Horizon, but I accept him as an accurate and helpful witness. 281. His evidence was very useful in terms of how Horizon works. The Horizon Issues are not about the actual design of the Horizon terminals in branch, and whether (say) this could be improved by having larger buttons or the screen layout adapted differently. In some of Mr Johnson’s evidence, the claimants’ counsel strayed into the areas of potential improvements to functionality in branch and design. These are not part of the Horizon Issues. The degree to which the evidence of fact affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions. Mr Dunks 282. Mr Andy Dunks is at IT Security Analyst at Fujitsu Services Ltd, and was the first Fujitsu witness to be called for the Post Office. He has been employed at Fujitsu on the Post Office account since 2002. He has working knowledge of the Horizon System. His evidence went to audit data extraction and the integrity of the data during that process. His evidence related to the process as it was when he made his statement, although given he had been involved for some 17 years his experience also related to Legacy Horizon and his evidence about data extraction covered that period too. As emerged in his cross-examination, he had also given a witness statement in the criminal trial of Mrs Seema Misra. 283. He agreed with the definition of data integrity put to him, which is that data integrity is the overall completeness, accuracy and consistency of that data which you can measure by comparing between sources. He said the 12 controls that he had listed in his statement explaining these he had compiled as a list from his recollection, and that he had not worked off a document to come up with the list in his statement. 284. Mr Dunks gave every indication, in the first part of his cross-examination, of being helpful and frank, and explained the process of data extraction and answered questions put to him openly. However, this encouraging start came to an end when he was asked about two curiously worded paragraphs in his statement. These stated as follows: “8. There is no reason to believe that the information in this statement is inaccurate because of the improper use of the system. To the best of my knowledge and belief at all material times the system was operating properly, or if not, any respect in which it was not operating properly, or was out of operation was not such as to effect the information held within it. 9. Any records to which I refer in my statement form part of the records relating to the business of Fujitsu Services Limited. These were compiled during the ordinary course of business from information supplied by persons who have, or may reasonably be supposed to have, personal knowledge of the matter dealt with in the information supplied, but are unlikely to have any recollection of the information or cannot be traced. As part of my duties, I have access to these records.” 285. Before I come to his evidence about this, it is obvious that the wording of paragraph 8 is almost that of a legal disclaimer (or a legally worded claim of accuracy, to be more precise), rather than a witness’ actual evidence. It would be very curious for a witness of fact to decide to put such a formally (and rather clumsily) worded paragraph in their witness statement. However, he was asked about these passages. He was asked what he meant by “the system”. The transcript records the following: “Q: What is the "system" there? Is that the system of the process of extracting audit data, or is it something else? (Pause). A. Good question. There's no -- I'm not sure what I was meaning by that, "
“Q: Again, pausing there, it is slightly confusing. Are you aware or not aware of any instances where that system was not operating properly? A. No, not really, no.” 287. He was asked if that part of the statement had been written for him by someone else, and he said he could not remember. He was also asked if it was a “Fujitsu party line” to put this in statements when it comes to extracting data and he said no, and that he was not aware of a Fujitsu party line. He accepted that gaps and duplicates in audit data were problematic, and he explained that they had to be investigated by audit support. It was clear to me that Mr Dunks considered audit data that had gaps and duplicates should not, in that form, be used. He said he would contact audit support if these occurred. 288. However, some of Mr Dunks’ evidence, in particular what he had to say about paragraph 8 of his own statement, proved in my judgment to be somewhat misleading. 289. He was shown a witness statement from Mr Gareth Jenkins from the Misra criminal trial in 2010. He said he had not seen this statement before. The statement put to him is some 8 years older, approximately, than his own statement. Mr Jenkins’ statement stated at the end: “There is no reason to believe that the information in this statement is inaccurate because of the improper use of the computer. To the best of my knowledge and belief at all material times the computer was operating properly, or if not, any respect in which it was not operating properly, or was out of operation was not such as to effect the information held on it. I hold a responsible position in relation to the working of the computer.” 290. The similarities between this passage, and the one in paragraph 8 of Mr Dunks’ witness statement almost 9 years later, are somewhat striking. Given he had not seen the statement before, there was no obvious explanation. His answer was as follows: “Q. And you're not sure why it would largely replicate your paragraph 8? A. No, I mean, we do have a standard witness statement that we produce for ARQs. When we supply ARQs we are sometimes asked for a witness statement to go through the process and verify as far as I'm aware that the data I supplied is accurate. Now, we use that quite a lot and it may actually be in that statement. Q. I see. So when I asked you earlier whether it was something of a Fujitsu party line you said you didn't think it was, but actually it looks as though it is on the basis of what you have just told me and on this document as well. A. It could be. It may be part of our standard witness statement that we supply.” 291. Mr Jenkins had given evidence in his witness statement from 2010 about duplicate entries in the audit data that had had to be removed. Mr Dunks was then taken to the witness statement of another Fujitsu witness in the Misra trial, called Penelope Thomas. She had given evidence about data extraction and listed a number of controls, which were remarkably similar to the ones that Mr Dunks gave evidence about (although Ms Thomas had omitted one). This is hardly likely to be a co-incidence, if Mr Dunks really had compiled his list of controls from memory or experience. She also had the identical passage in her witness statement as Mr Dunks’ paragraph 8. 292. Mr Dunks also said in respect of this, which in my judgment painted a rather different picture to that portrayed in his witness statement, the following: “A. The statement again is to verify the integrity of the data once extracted and given. We don't control what's in the data. Our process is about extracting it and securely passing it over to the Post Office. It's not our concern of what's in the data. Q. It's not your concern what's in the data? A. No, it's what we're -- we're process-driven to extract certain types of data for certain requests. Q. And so whether or not that might match another record or not, or replicate or duplicate or have gaps, that's not part of your remit, that's not really part of your concern? A. No, it's not. Q. I see. So reading that and reading the paragraph 8 as it was in your witness statement, it doesn't give much comfort to somebody that's then trying to rely on ARQ data as being a gold standard to compare and investigate anomalies, does it? A. Possibly not.” 293. The reference to “gold standard” was the expression that had been used by Dr Worden in his report to describe the quality of the audit data that was held by Fujitsu. It should be a very great concern to anyone tasked with extracting audit data, should that data prove to have gaps and duplicates in it. This is because gaps and duplicates in the data affects the accuracy of that data. I consider that such a cavalier approach to whether audit data has gaps and duplicates, as evidenced by Mr Dunks’ saying that such things were not part of his concern, to be entirely contradictory to a statement verifying the accuracy of such audit data. 294. My findings in relation to this are as follows: 1. Mr Dunks expressly sought to mislead me by stating that there was no “Fujitsu party line” when it came to the contents of drafting witness statements about audit records for legal proceedings. There plainly is; it was used in the Fujitsu statements in 2010 and it was used by him in his statement for the Horizon Issues trial. 2. The passage included in paragraph 8 of his statement is plainly a standard form of words, and reads as though it is a legally drafted assertion of accuracy. The witness’ assertion that the system was at all material times working properly, or if it wasn’t, this did not affect accuracy, does not sit consistently with Mr Dunk’s acceptance of gaps and duplicates in audit data. 3. That Mr Dunks did not really know very much about paragraph 8 of his own witness statement was confirmed by the way he was puzzled when he was asked about it. 4. However, Mr Dunks’ acceptance of standard wording provided by others (probably within Fujitsu) is less important than the fact that he originally tried to mislead the court. 5. I do not accept that he identified the 12 controls in his statement from his recollection. He must have been working off another document, which he was not prepared to identify. I draw this conclusion because of the marked similarity with the controls included in Mrs Thomas’ statement from 2010; the wording of the controls themselves; and the fact he was prepared to mislead me about the paragraph 8 point. 6. In re-examination he asserted that the statements in his paragraph 8 were true, and that he would not have signed his statement had this not been the case. I find that he cannot have known whether those statements were true or not. There is no evidence of any steps he took to check whether or not they were true or not. I find they were simply standard sentences supplied to him, and I find that he signed his statement without any independent knowledge of whether they were true or not. 7. I find that passage in his witness statement simply to be meaningless assertion. 295. He said that he was unaware of Post Office internal concerns that there was “a lack of system audit trail” in a document of22 October 2016 . I find his lack of knowledge about this surprising, given his role. 296. I found Mr Dunks very unsatisfactory as a witness. He was both plainly aware of the Fujitsu “party line”, or corporate position, regarding the words asserting accuracy of audit data, and he was very anxious to keep to it, whilst initially denying that there was one. He sought to mislead me about both his paragraph 8 wording, and the way he had compiled his list of controls. The degree to which the evidence of fact affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions. Mr Godeseth 297. Mr Torstein Godeseth is employed by Fujitsu and is the Chief Architect on the Post Office account. He had provided three witness statements, as had Mr Parker. After graduating from Oxford with a degree in physics in 1974, he worked for Rolls Royce as a combustion engineer and then joined the Royal Navy in 1977 as an Instructor Officer. Whilst in the Royal Navy, he started his career in IT working in systems programming. It will be remembered that in the 1970s computing and IT was a relatively young field, and Mr Godeseth has therefore been involved in that industry since its relative infancy. He left the Royal Navy in 1981 and worked in systems programming and technical support for the IT at Forward Trust Ltd. He joined the Post Office IT department in November 1987. Initially this was to work on a project to introduce technology into Post Office branches. 298. He was working with the Post Office as a technical advisor when the Post Office and Benefits Agency procured the Horizon system originally. He remained involved in different roles, including being outsourced from the Royal Mail IT department to Xansa in 2003, and contracted to the Post Office to act as the technical adviser interfacing with Fujitsu amongst others. Although an independent contractor, he worked between 2005 and 2010 with the Post Office as the technical advisor on IT projects including the change from Legacy Horizon to Horizon Online. He joined Fujitsu as a full employee in 2010. He described himself in cross-examination as the designer of Horizon. 299. He had a greater knowledge of Horizon Online than of Legacy Horizon, although given his involvement in Horizon from its early days he had significant knowledge about both. Horizon was initially, in the late 1990s, called the Pathway project and was a joint effort between the Post Office and the Benefits Agency, including a way to computerise the payment of benefits to those entitled to them, through payments to benefits claimants which were to be made through the Post Office by means of a swipe card method. This project ran from approximately 1996 to 1999 and was to be provided by ICL (which Fujitsu partly owned and later fully acquired). At some point during the development of this project the Benefits Agency withdrew, and what had been the Pathway project, essentially a tri-partite venture (the parties being ICL, the Post Office and the Benefits Agency) became Horizon, now called Legacy Horizon. 300. Although this litigation is not about the way that the Horizon system was designed, it is about its functionality and robustness in use, Mr Godeseth said that it was a specific Post Office decision not to have any dispute button/function for SPMs built into the Horizon system. 301. He said: “I think the basic argument was that disputes -- we wanted the flow of data through the system as quickly as possible because that keeps our books tidy and it was an inference that there was always the -- you had to press a button to take things through, but then you would pick up the phone to NBSC and say that wasn't right.”
“In addition, in common with many elderly systems that have been subjected to a succession of major changes, it [ie Legacy Horizon] has become increasingly difficult to make those changes, and expensive to operate." This was put to Mr Godeseth: “Q: Now, that's a fair description of how the system originally was perhaps designed jointly with the DSS at the beginning and launched and then over the years, between then and 2008, there have been lots of sort of bolt-ons and additional things that have been changed on the system, haven't there? A. Yes, I think that's fair. The major one probably in my mind would be banking.”
“Q: You realised for the first time that it was possible to inject or insert a transaction with a counter position less than 32 when Mr Parker was preparing his second statement? A. Correct. Q. And you knew that that was a contentious issue in this litigation, yes? A. Yes.” 320. And further: “Q. Now, it would be possible, would it not, to use a counter number of 1, or 2, or 3? A. It would. Q. And if that counter number was a counter number actually in use by the SPM, it would appear to the SPM, from the records they could see, that it was a transaction which had been done in their branch, by them or their assistants? A. Yes .”
“The following metrics, to be reported monthly, will be used to measure effectiveness of the process and drive performance of the process and overall service in general: Number and impacts of incidents occurring before root problem is identified and resolved Number of repeat incidents following corrective action Number of problem records arising from pro-active actions and trend analysis Number of changes arising from proactive actions Percentage of problem records without an action plan Average length of time to resolve problems Number of incidents closed without a KEL The number of Problem Records arising from Managed Changed (MSC, CP) activities The number of Problem Records arising from the implementation of new services/major releases.” 325. However, when the claimants had sought to obtain reports that would be expected to exist, which included or demonstrated the reporting against these bullet points which Fujitsu’s own document stated were required, and which were supposed to be prepared every month, Fujitsu stated (through the Post Office’s solicitors) that “Fujitsu believes that it does not record problems in such a way that would allow this to be determined without retrospectively carrying out detailed analyses” and that it would require “a disproportionate effort and cost” to provide these. Mr Godeseth also gave written evidence that the reporting system identified in Fujitsu’s own document had not been implemented and that the records did not even exist. The passage in his statement said: "I have spoken to my colleague Steve Bansal, Fujitsu's senior service delivery manager, who has informed me that the Post Office account customer service problem management procedure document was introduced by Saheed Salawu, Fujitsu's former Horizon lead service delivery manager and that Saheed Salawu left the Fujitsu Post Office account in around February 2013, before the new procedure had been implemented. I understand from Steve that Saheed Salawu's replacement did not wish to implement the changes and therefore the records referred to by Mr Coyne in paragraphs 5.157 to 5.159 of his report do not exist, as we continued to follow the previous existing reporting methodology .”
“The following transaction tables have been granted INSERT privileges to OPS$SUPPORTTOOLUSER. The transaction correction statement is only allowed to insert into these tables.”
"Each document is associated with a single SQL statement which made a database correction. There are two different types of correction shown in the files - the SQL statements for each are of the form: "1. Update OPS$BRDB.brdb_rx_recovery_transactionsSET_ settlement_complete _time stamp = ..."
“After discussion with Gareth Jenkins, the suggested correction is to negate the duplicate transfer out by writing 2 lines to the BRDB_RX_REP_SESSION and BRDB_RX_EPOSS_TRANSACTIONS tables, with: 1) Product 1, Quantity 1, Amount 4000.00, Counter mode id 7 (TI) 2) Product 6276, Quantity -1, Amount -4000.00, Counter mode id 7 (TI) This should be done using the Transaction Correction tool. An OCP approved by POL will be needed.” 349. This concerns Horizon Online, due to the references to OCP and OSR, and shows extensive efforts being made at Fujitsu to try to understand why this has occurred, and general doubt that the efforts to reproduce the fault in testing has worked. Some at Fujitsu wanted the priority to be downgraded; other entries resisted this, shown by some of the entries such as: “What is missing from this Peak is an explanation of the events in terms of the requests, how they were ordered and when any was committed. Only then can we qualify the priority. The assumption is that we have a fix. The facts are – 1: A settlement request to timed. 2: A retry of request timeout occurred. 3: According to the DB entries both later succeeded. Now unlike other reconciliation Peaks this stands out….. We can’t reduce the priority unless we understand what is going on.” 350. The PEAK is lengthy and demonstrates a degree of frustration within Fujitsu at their own failure to get to the root cause. One entry on24 March 2010 states inter alia “…..this shouldn’t have happened”
“If this in fact happened as written down here [ie in the PEAK] then it would have had to have been a bug in Oracle and I certainly don't remember any such bug.” 352. The PEAK also shows computer script being used, both in an attempt to correct, and understand and analyse, what occurred. On23 April 2010 an entry states: “I think we have done as much as we can on this one. In conclusion although we haven’t been able to totally explain the behaviour, the risk of this type of PEAK occurring again has been minimised in live due to a change of behaviour in the BAL with respect to transactions…..This may now be marked as a duplicate of PC0194893”
“I am sending this call back with Response Rejected. Closing a call as 'Duplicate Call' results in a black mark against me. It basically means that I should not have sent the call over since the same problem has already been sent over in a previous call. PC0195561 (duplicate transfer of 4000.00 cash) may have been caused by the same underlying fault as PC0194893 (banking reconciliation), however I could not have been reasonably expected to link the 2 calls and take the decision that it was not necessary to send PC0195561 over for further investigation. Please close this call with category 'Advice After Investigation' [End of Response] Response code to call type L as Category 52 -- Pending -- Response Rejected Response was delivered to Consumer.” 354. Eventually KEL cardc262s was updated with the information in the PEAK on4 May 2010 and the call was closed. One of the final entries states that day “Category 60 – Final – S/W Fix Released to Call Logger”. “S/W fix” means software fix. The call is closed the same day. This is 2 months after the incident. The PEAK also referred to other PEAKs. Mr Godeseth was asked about those too. PEAK 0195962 records that the transaction tool has “been used in live”
“ Due to a system fault , the branch did a Transfer Out of£4000 and a corresponding Transfer In of£8000 Justification: Correct a loss of£4000 at the branch due to a system fault When: Planned for10/03/2010 16:00 with a duration of 30 minutes Extra detail: The Transfer In details were incorrectly doubled up when they were written to the BRDB. This needs to be corrected using the Transaction Correction tool.”
“In this situation the symptoms, as I'm reading them, are that because there was a bug in Horizon and this is in the new system, it was pretty early days of the Horizon Online, because there was a problem, we had something coming through which got through the journal filter but then failed at the branch database and so therefore, as far as the branch database is concerned, it has not happened . MR JUSTICE FRASER: I understand that. I think what Mr Green is putting to you is that this shouldn't have got past the journal filter. A. It's a bug , so certainly the way that the system should have worked, a JSN -- the same JSN coming up would be just a simple repeat of the message for -- because there was some sort of glitch .”
“A. Somebody may have been trying to look for a duplicate JSN entry. I can't really comment on what the guy was doing at the time who was trying to investigate. I am simply asserting that barring an Oracle bug, which would have been huge, you cannot have two entries with the same JSN.” 359. In my judgment, the Horizon Issues are sufficiently wide and costly to resolve concerning the Horizon System, both Legacy and Online, without widening them to include some attempt at including specific bugs within Oracle too. The Oracle Corporation is not even a party, and has had no opportunity to provide evidence or submissions. I simply recount Mr Godeseth’s evidence as he attempted to explain the PEAK. He did, as can be seen, refer to potential bugs in Oracle as an explanation for something that he believed to be puzzling. He accepted it was a bug and/or “a glitch”
“ We are unable to correct the system figures safely . We can however provide accurate figures for what should have been in the Final Balance for BB, to enable POL [ie the Post Office] to make the correction perhaps by using a Transaction Correction. POL need to make a decision on whether they are able to correct the problem in this way, however we do not see any other alternative . Corrective action should be taken before 11th January when the branch is due to roll into TP10. The cause of the problem is unknown and is under investigation.”
“If we get to the problem before the office is rolled we are able to change objects in the messagestore to reset the stockunit back to the CAP (TP) rollover trailer. The PM can then rollover. PM should get a large shortage which cancels out the large gain. We don't want to be having to do this as making manual changes to the messagestore is open to error and each time we have to seek authorisation from POL to make the changes. If we get to the problem after the office is rolled (as in this call) then we are unable to correct the system figures safely. Its not been decided how we get the PM sorted out. All in all, we want this fixed asap.”
“This is due to a single SC line written for$1000 (£484 ) with no settlement in the middle of two RISP transactions. On call PC0151718 the harvester exception was corrected and now the transaction for the day don't zero, hence this issue with the incomplete summaries report. Am currently retrieving the messagestore for this branch, we will then be inserting a new message on the counter to remove the effects of this. OCP 17510 has been raised.” 365. Again, this was being done to correct the effect of what had occurred, rather than discovering what had led to the issue in the first place. In two places in the PEAK, the following entry appeared: “**Again, this may also have caused a receipts and payments error, can EDSC please confirm whether this is a gain or loss at the counter and the amount.**” 366. Mr Godeseth had been asked to look at this PEAK in advance (his evidence went over two days and this was agreed by the parties), and therefore he had been given a chance to familiarise himself with its contents, and those of an associated KEL. The PEAK stated "Worth noting that the branch did not have any issues with the mismatched transactions because this was fixed before they did the roll. The branch is not aware of this and it's best that the branch is not advised ." (emphasis added) 367. When it was put to Mr Godeseth that this showed that on not all occasions were SPMs advised of impacts on their branch accounts of particular problems, he said that was “a fair inference”
“A single SC message 183227-7-1101211 was written in error on 26th November at 12:43:17, selling 1000 US dollars, with no corresponding settlement line. To remove the effects of this message at both the branch and on POLFS, we will insert a new message to negate the effects of the original message. Justification: If the change is not made in the counter messagestore (before the stock unit is balanced on Wednesday), the branch will have an unexpected gain of£484 (or thereabouts - depends on exchange rate), and a receipts and payments mismatch. This gain would have to be resolved at the branch. There would also be an inconsistency between the branch and POLFS to be resolved. By correcting the problem locally, the branch may not be aware of the problem , and there will be no inconsistency between the branch and POLFS.”
“The message will include a comment to show it has been inserted to resolve this problem (this will not be visible to the branch). This change will first be applied to a copy of the messagestore within the SSC environment, and the stock unit then rolled over to make sure there are no unexpected consequences. Neither the new nor the old message will be included in data sent to POLFS. Gary Blackburn (POL) is already aware of this issue.”
“Extra detail: The original message had ProductNo:5129, Qty:1, SaleValue:484, PQty:1000. The new message will have Qty:-1, SaleValue:-484, PQty:-1000, with other attributes (including exchange rate) as before.” 370. The specific message that was inserted into POL FS, as shown in the OCR, was as follows: “Extra detail: This OCR is being raised so that EDSC is authorised to amend the txn details for branch 183227 and insert these into tps_pol_fs_summaries_incomp table on the host. Comments Andy Keil (POA SSC Support) wrote at12/12/2007 15:07: Updated POLFS feed for branch 183227 product 5129 mode SC with SaleValue=1014.73 and PQty=2080”
“The counter problem which caused the first issue has been corrected by inserting a message into the messagestore, for equal but opposite values/quantities, as agreed with POL (OCP 17510). As a result of this corrective action, the net effect on POLFS is zero, and POLFS figures are in line with the branch. POLMIS received both the original message and the corrective message. Once the problem was corrected, there should have been no impact on the branch. However it has been noted that the stock unit BDC had a loss of$1000 , which was generated after the correction was made. We have already notified Gary Blackburn at POL (email attached). This appears to be a genuine loss at the branch, not a consequence of the problem or correction .”
“Summary for development: A single SC line was written for$1000 (£484 ), with no settlement, in the middle of two RISP transactions. The line was missing some AdditionalData so it wasn't harvested properly, but the main problem was the lack of settlement. POL authorised us to insert an equal but opposite message, to prevent a discrepancy (in theory anyway) and to avoid problems on POLFS. Please note that this is exceptional and must not be seen as a convenient avoidance in place of a fix. Subsequent investigations showed that a second packed pouch barcode had been scanned, before the receipt for the first had finished processing. See the audit log / messages on counter 7 26th Nov at 12:43 / 12:44. PC0147357 is already with development for what I think is a similar problem - Mode reverts to what it was prior to RISP. Please also see update on that call. Routing to EPOSS-Dev via QFP. [End of Response] Response code to call type L as Category 40 -- Pending -- Incident Under Investigation.”
“Q. Yes. So what they are talking about here, what this change here is, is a change being made by using the TIP repair tool into the TPS, correct? A. Right, so this doesn't relate to the creation of the message then. Q. It doesn't relate to the branch accounts, Mr Coyne, does it? This is an OCR which involves an exercise -- well, the use of the TIP repair tool to change data that is in the TPS system, yes? A. Yes, but the PEAK refers to the insertion of a message into the messagestore.” 381. The critical point, that the explanation skipped over, was what the message that was inserted in fact was, in actual terms. Had that message been produced, it could have been looked at, and deciphered. Understanding such a message is not an insuperable task beyond mortals. The second point is that the PEAK refers to a KEL, namely KEL obengc3120K. At the beginning of the trial, I asked for hard copies of all PEAKs and KELs put to witnesses to be prepared in hard copy too, so that I had a working file that I could use during the trial for PEAKs and KELs already referred to (in addition to the three OPUS trial bundle screens that I had to work from). This was done. That KEL is at tab 21 of volume 2 of the file entitled “PEAKs and KELs referred to in Days 5-8 and 12”
“The first is where all five SC transactions missing core data as described in the above-mentioned KEL." This means obengc3120K, referred to at the beginning of the PEAK. And the “Second is absence of equal but opposite 2 (ie settlement) lines. See PC0152014 for a similar problem and how problem was resolved." “There are two sides to the problem relating to these txns. The first is where all five SC txns missing core data as described in the above mentioned KEL. Second is absence of equal but opposite [i.e. settlement] lines. See PC0152014 for a similar problem and how problem was resolved. For the first problem, I have used the TRT [the transaction repair tool] to insert the missing data i.e. Region, Margin, Margin Product and EffectiveExRate.”
“As per the previous PEAK comments, the role 'APPSUP' is extremely powerful and should only be used under extreme circumstances and under MSC supervision . As such the Branch Database design was that 3rd line support users should be given the 'SSC' role, which is effectively read access, ie. 'select_any_table + select_catalogue'. SSC team members should only have to [access] BRSS for normal support investigations, unless the information has not replicated in time. SSC should only given the optional role 'APPSUP' temporarily (by Security Ops authorisation/emergency MSC) if required to make emergency data amendments in BRDB Live. It is a security breach if any user write access is not audited on Branch Database, hence the emergency MSC for any APPSUP role activity must have session logs attached under the MSC. Host-Dev previously provided scripts, such as the Transaction Correction Tool, are written to run under the SSC role and also write to the audit logs. SSC users created on BRDB should only have the SSC role, and the user creation script should be amended by Host-Dev to reflect this. A separate script giving/revoking emergency MSC access via APPSUP can be delivered, logging this to the hostaudit directory. In parallel Host-Dev should investigate any Host-Dev delivered script to ensure they are all executable by the SSC role. SSC should investigate any of their own scripts to ensure they have sufficient permissions under the SSC role, taking into account they should primarily perform their work on BRSS. Any day to day scripts should not access BRDB directly . Any scripts needing more than the SSC role should be questioned, except those that would run under MSC APPSUP. Once the investigation is complete, all BRDB SSC users with APPSUP should have the role removed by ISD (via MSC) and ensure they do have the SSC role. If anyone is in disagreement with the above course of action, then I'll set up a meeting with yourselves and Security when I'm down in BRA01 next week.”
“The Business Impact has been updated: 1. Cost: There is currently no "cost" to this issue. The users affected have more access than is required. 2. Perceived Impact: The customer is not aware of this problem or change. 3. Scope: No actual impact/incidents of problems relating to this issue have been experienced yet (and not expected)”. (emphasis added) The customer was the Post Office. The “issue” referred to was Fujitsu SSC users having had far greater access, namely the powerful APPSUP permissions, on more than the intended basis, which was supposed to be in extreme circumstances only and temporarily. This is shown in an entry for1 February 2011 by Mark Wright who stated “….SSC users have the APPSUP role” and “When we created SSC users for BDB/BRS etc. we used “appsup” as that is what SSC have always been and what they migrated as on Horizon databases.”
“Unfortunately development write their scripts explicitly to use SSC. So I think we're stuck with it unless they deliver new scripts (which would not be a popular or quick option). When we go off piste we use appsup . Can we have both??”
“The main area we would encourage management focus on in the current year is improving the IT governance and control environment. Within the IT environment our audit work has again identified weaknesses mainly relating to the control environment operated by POL’s third party IT suppliers . Our key recommendations can be summarised into the following four areas: Improve governance of outsourcing application management Improve segregation of duties within the manage change process Strengthen the change management process Strengthen the review of privileged access”
“This is an Oracle supplied role for use by database administrators (DBAs). Lots of privileges are granted to this role so users have the ability to update/delete/insert into any of the Branch database tables .”
“Granted the ability to execute the following executable functions: (a) OPS$SUPPORTTOOLUSER.PKG BRDB TXN_CORRECTION —framework to allow the user to insert fully audited balancing records into a BRDB transaction table (made against node ID 99).” 397. Both of these are powerful roles. Mr Godeseth was asked specifically about the DBA role explained in [395] above. His specific answers merit quotation verbatim: “Q. You would agree that those people have the role which allows them privileges to update, delete, or insert into branch database tables whether they are using the correction tool or not? A. Those people could log on to the database and do an awful lot of damage. Q. And the only audit of that that we have prior to 2015 was log on and log off; that's correct, isn't it? A. Correct. ”
“I have not examined the privileged user logs, but based on my experience my expectation is that these uses of APPSUP, or at least the vast majority, are for support work that does not involve changes to transaction data. I cannot recall any cases in which it has been used to change transaction data, but I cannot state unequivocally that there are no circumstances in which it has ever happened.” 399. Mr Godeseth said in his written evidence that “Privileged users can, theoretically, inject, edit or delete transaction data in Horizon Online. As far as I am aware, this has never happened.”
“SMC: If the event is seen at a multi-counter office during the working hours of the PO, or up to 18:00 on a weekday (in case they are balancing out-of-hours), RING THE OFFICE AND GET THEM TO REBOOT the eventing counter. If they are in the process of balancing, it is strongly advised that they reboot before continuing with balancing as they are at risk of producing an incorrect balance . Warn the PM that if transactions appear to be missing, they should not be re-entered - they will become visible after the counter has been rebooted. If a reboot/Cleardesk does not resolve this problem, send the call over for further investigation - SSC can rebuild the messagestore on the affected counter.” (Block capitals in original; emphasis by underlining added) 403. The risk of a branch producing an incorrect balance as a result of this problem with Riposte is clearly recorded. An incorrect balance is another way of referring to discrepancies in branch accounts, a central issue in this litigation. 404. The Callendar Square issue or consequences (now admitted by the Post Office to be a bug) were discovered in 2005 at the Callendar Square branch (hence the name). It is also sometimes called the Falkirk bug, but I shall refer to it as Callendar Square. It undoubtedly caused discrepancies in the accounts. The SPM in question had support from their Area Manager and the documents demonstrate the following, from the Area Intervention Manager Visit Log: “This office had severe problems balancing on Wk 25, resulting in a shortage of£6,414.46 . After checking various reports I am satisfied that the error is made up of:£3,489.69 – Transfers£2,870 – Giro Deposits£54.52 – unidentified (however due to all the coming and going with re- keying entries, then this could come back as an error). The Spmr claims that there was a Horizon software problem on 14.09.05 from 15.30 onwards. This was picked up when a member of staff noticed that a transaction, which had been taken by another member of staff, had not been entered onto the system, so therefore she put the transaction through. She checked at the time with her colleague who said that she thought she had put it through already however she accepted that she could have made a mistake. Following on from that, it was picked up that other giro business deposits that had been entered had not come up on the system, so they were re-keyed. There was also a problem with transfers from one stock to another, in that they had doubled up. The Spmr made several telephone calls to the NBSC, telling them about his problems and he was advised to carry on with balancing and produce his Cash Account. Whilst doing this a warning came up, however the NBSC told the staff to continue to roll over. The result was that the office balanced£6,414.46 short. The Spmr spoke to the Horizon Support Centre (ref E0509150123) who investigated and agreed that there had been a navigation problem that had now been rectified. They told the Spmr that they would report to NBSC that they had identified and rectified the problem and that the amount could be held in the suspense account. However, as part of the shortage relates to transfers, and no error notice will be issued, then the Suspense Account Team are not prepared to authorise the entry. I telephoned The Suspense Account Team (Ann Wilde), who told me that checks could be made with Girobank after next Wednesday, and if that shows that duplications have been made, then they will authorise the amount to be moved to the suspense account, until the office receives an error notice. However, Ann stands by what she said about the transfer problems, and that they would not move this amount to The Suspense Account. I went back to the Horizon Support Centre and spoke to a supervisor (Ken). He said that the call had now been closed as the problem had been rectified. I asked what was to happen about the resulting shortage and he referred me back to NBSC, who they said would do various checks. I then contacted NBSC, spoke to Rob Hughes and told him the story – he said he would put a call through to Service Support. No follow up was received from Service Support regarding this call.”
“1/ This problem is the root cause of the reconciliation error closed in PC0126042 2/ Presumably the root cause is deemed to be software not hardware 3/ The Postmaster has a workaround in place which is not to duplicate transactions (e.g. Transfer In) just because the original attempts were successful but not showing on all nodes 4/ POA CS MSU have a workaround in place which is that if 3/ above is not followed & PC0126042 reoccurs, a BIMS will be issued advising POL to issue a Transaction Correction 5/ There is no SLT for software fixes as they are delivered based upon the priority or severity of the issue and could remain open until both businesses decide a fix is necessary or the work around is adequate.”
“Issue Raised: PM WANTS TO SPEAK TO SANDRA MCKAY URGENTLY REGARDING SERIOUS PROBLEMS WITH HORIZON IT IS REPEAT PERFORMANCE AS TO WHAT WAS HAPPENING BEFORE AND HSH FAILING TO PICK IT UP Response by SPM: Telephoned the office and Allan said that he was having problems again with transfers. He has contacted the Horizon helpdesk who have subsequently come back to him to say that there is no system problem and that he should contact NBSC . He did this and from what I can understand the NBSC have told him that he is trying to balance on two different terminals. Allan disputes this and is adamant that there is a system error.” (Block capitals in original; emphasis by underlining added) 410. The statement that “there is no system problem” which is told to the SPM evidently conflicts with what the PEAK at [407] had recorded, that “presumably the root cause” was the software. 411. Emails in February 2006 to and from Anne Chambers stated expressly the following: “…. I notice that in the early guise of this problem in the call it states the PM as Female: Wed12 October 2005 17:39 by UK956078 / HSH1 Saved: Wed12 October 2005 17:39 Pm was trying to transfer£2490 from node 7 onto node 2. She states that she has accepted the transfer on node 2 but the system is not showing this. On node 7 it is showing pending transfer but it is not showing on node 2. It appears on her transfer sheet as completed. At the bottom of this email re a magical£43k appearing and disappearing the PM is Male He reports: You may recall that in September the above office had major problems with their Horizon system relating to transfers between stock units. The Spmr has reported that he is again experiencing problems with transfers, (05.01.06) which resulted in a loss of around£43k which has subsequently rectified itself. I know that the Spmr has reported this to Horizon Support, who have come back to him stating that they cannot find any problem. Clearly the Spmr is concerned as we have just spent a number of months trying to sort out the first instance and he doesn't want a repeat performance. He is convinced that there is something wrong with his Horizon kit . I would be grateful if you could investigate this and give him any support that you can. I'm due to visit the office tomorrow to have a look at his paperwork and discuss the situation with him. So apologies for the long windedness but I have been given this by Liz as a problem so: 1. Is there a problem at this branch? is it Horizon kit or is there an issue with staff there ? 2. If there is an issue is this S90 release the cure? how confident are you/we it will fix the problem? 3. S90 counter release due week 4th March. Getting Sarah to check if this site is in the pilot 24th or just part of the general release 4th March. Appreciate your comments please.”
“Haven't looked at the recent evidence, but I know in the past this site had hit this Riposte lock problem 2 or 3 times within a few weeks. This problem has been around for years and affects a number of sites most weeks , and finally Escher say they have done something about it. I am interested in whether they really have fixed it which it why I left the call open - to remind me to check over the whole estate once S90 is live - call me cynical but I do not just accept a 3rd party's word that they have fixed something! What I never got to the bottom of, having usually had more pressing things to do, was why this outlet was particularly prone to the problem. Possibly because they follow some particular procedure/sequence which makes it more likely to happen ? This could still be worth investigating, especially if they have continuing problems, but I don't think it is worthwhile until we know the S90 position. Please note that KELs tell SMC that they must contact sites and warn them of balancing problems if they notice the event storms caused by the held lock, and advise them to reboot the affected counter before continuing with the balance. Unfortunately in practice it seems to take SMC several hours to notice these storms by which time the damage may have been done .”
“I’ve been asked about the issue at Callender Square, Falkirk (I’m not quite sure about the spelling) that came up at the Castleton Trial. I thought I’d better keep you in the loop on this. I’ve now dug back into the archives to provide the following summary: 1. The problem occurred when transferring Cash or Stock between Stock Units. Note that West Byfleet does operate multiple Stock Units so the issue could have occurred. 2. It manifests itself by the Receiving Stock Unit not being able to “see” the Transfer made by the “sending”
“NB many other branches had multiple events, preventing replication, but these are the majority of those which came to Peak , having either reported a problem or it caused a reconciliation report entry. From Sept 2005, cash accounts were replaced by branch trading statements and the TPSC256 report was no longer populated. I can't remember how we then knew about receipts and payments mismatches and if we would have picked up on further issues.”
“What is the issue? Discrepancies showing at the Horizon counter disappear when the branch follows certain process steps, but will still show within the back end branch account. This is currently impacting circa 40 Branches since migration onto Horizon Online, with an overall cash value of circa£20k loss. This issue will only occur if a branch cancels the completion of the trading period, but within the same session continues to roll into a new balance period. At this time we have not communicated with branches affected and we do not believe they are exploiting this bug intentionally. (emphasis added) The problem occurs as part of the process when moving discrepancies on the Horizon System into Local Suspense. When Discrepancies are found during Stock Unit rollover into a new Transaction Period, then the User is asked if the discrepancy should be moved to Local Suspense. If the branch presses cancel at this point the Discrepancy is zeroed on the Horizon System. Note at this point nothing into feeds POLSAP and Credence, so in effect the POLSAP and Credence shows the discrepancy whereas the Horizon system in the branch doesn't. So the branch will then believe they have balanced. If at the next screen the rollover is completely cancelled, then no harm is done. However if the Rollover is re-attempted at this point, the rollover will continue without any discrepancy meaning Horizon doesn't match POLSAP or Credence. This has the following consequences: • There will be a Receipts and Payment mismatch corresponding to the value of Discrepancies that were "lost" Note the Branch will not get a prompt from the system to say there is Receipts and Payment mismatch, therefore the branch will believe they have balanced correctly. • When the Branch begins the new Branch Trading period the discrepancies will show at Zero, however the Receipts and Payment mismatch will carry over into the next period. Impact • The branch has appeared to have balanced, whereas in fact they could have a loss or a gain. • Our accounting systems will be out of sync with what is recorded at the branch • If widely known could cause a loss of confident in the Horizon System by branches • Potential impact upon ongoing legal cases where branches are disputing the integrity of Horizon Data • It could provide branches ammunition to blame Horizon for future discrepancies. Identifying the issue and forward resolution The Receipts and Payment mismatch will result in an error code being generated which will allow Fujitsu to isolate branches affected this by this problem, although this is not seen by the branches. We have asked Fujitsu why it has taken so long to react to and escalate an issue which began in May. They will provide feedback in due course. Fujitsu are writing a code fix which stop the discrepancy disappearing from Horizon in the future. They are aiming to deliver this into test week commencing 4 th October. With live proving at the model office week commencing 11 th October. With full roll out to the network completed by the 21 st of October. We have explored moving this forward and this is the earliest it can be released into live. The code fix will on stop the issue occurring in the future, but it will not fix any current mismatch at branch. Proposal for affected Branches There are three potential solutions to apply to the impacted branches, the groups recommendation is that solution two should be progressed. SOLUTION ONE - Alter the Horizon Branch figure at the counter to show the discrepancy. Fujitsu would have to manually write an entry value to the local branch account. IMPACT - When the branch comes to complete next Trading Period they would have a discrepancy, which they would have to bring to account. RISK- This has significant data integrity concerns and could lead to questions of "tampering" with the branch system and could generate questions around how the discrepancy was caused. This solution could have moral implications of Post Office changing branch data without informing the branch. SOLUTION TWO - P&BA will journal values from the discrepancy account into the Customer Account and recover/refund via normal processes. This will need to be supported by an approved POL communication. Unlike the branch "POLSAP" remains in balance albeit with an account (discrepancies) that should be cleared. IMPACT - Post Office will be required to explain the reason for a debt recovery/ refund even though there is no discrepancy at the branch. RISK - Could potentially highlight to branches that Horizon can lose data. SOLUTION THREE - It is decided not to correct the data in the branches (ie Post Office would prefer to write off the "lost" IMPACT - Post office must absorb circa£20K loss RISK - Huge moral implications to the integrity of the business, as there are agents that were potentially due a cash gain on their system.” (bold present in original) 429. I find this to be a most disturbing document in the context of this group litigation. It is a 2010 document and issues between the Post Office and many SPMs concerning the accuracy of Horizon had, for Legacy Horizon, gone on for a decade (2000 to 2010) and these continued under Horizon Online (introduced in 2010). Under “Impact”, some of the bullet points incorporate a summary of these issues. “• The branch has appeared to have balanced, whereas in fact they could have a loss or a gain. • Our accounting systems [ie Horizon or the Post Office’s] will be out of sync with what is recorded at the branch • If widely known could cause a loss of confident in the Horizon System by branches • Potential impact upon ongoing legal cases where branches are disputing the integrity of Horizon Data • It could provide branches ammunition to blame Horizon for future discrepancies.”
“1. The purpose of this note is to document a request that we have had from Post Office in terms of presenting details of what happened as a result of a bug in HNG-X in September 2010 which caused a Receipts and Payments mismatch and also resulted in Discrepancies being lost. The background to this is the fact that there was a BBC documentary broadcast on Monday 7 th February 2011 reporting on postmasters being unhappy about being pursued for losses by Postmasters on Horizon. It should be noted that the issues described here relates to HNG-X (Horizon Online) and that the implementation of the accounting mechanisms in the two systems is totally different (but they do produce the same reports and support the same business process).”
“It should be noted that as Discrepancies are normally Losses, then a Lost Discrepancy would normally work in the Branches favour and so there is no incentive for the branch to report the problem. Also if we do amend the data to re-introduce the Discrepancy, this will need to be carefully communicated to the Branches to avoid questions about the system integrity .”
“For each of the Dalmellington and Callendar Square bugs, could the parties please indicate (by means of an agreed fact but only if that is possible, if not each party’s contended for answer) their answers to the following two questions: 1. The date when the existence of each of these was communicated by the Post Office to SPMs; 2. A trial bundle reference to a contemporaneous document, email or communication from the Post Office demonstrating question 1.” 436. The parties could not agree on a date when the existence of these two bugs – Callendar Square being an acknowledged bug in the Bug Table dealt with below (from the 2 nd Joint Statement of the two experts) and Dalmellington, which is now accepted to be a bug – was communicated to SPMs. The claimants’ answer to the two questions was the same for each bug, namely that there was “no date in any contemporaneous document from the Post Office in the trial bundle showing that such communication in fact took place and identifying the actual content of any such communication.”
“It has never been Post Office’s case that, once it was discovered, the existence of the Callendar Square bug was communicated by Post Office to all SPMs or communicated to SPMs who had been affected. Post Office's case is that it was unnecessary and would have been inappropriate to do so.”
“It has never been Post Office’s case that, once it was discovered, the existence of the Dalmellington bug was communicated by Post Office to all SPMs or communicated to the SPMs who had been affected. Post Office's position is that it was unnecessary and would have been inappropriate to do so.”
“ In April 2011 a problem was found with the archiving strategy related to Stock Units that have been deleted in a Branch. A consequence of this is that some changes were made to the archiving strategy on 3 rd July 2011. An unintended consequence of this change was that any Branch that deleted a Stock Unit at the end of 2010 which had local suspense transaction in that Stock Unit before it was deleted were left in the table used for constructing the BTS. This meant that as Trading Periods cycle around each year, these BTS records became visible in 2011 when the same Trading Period was reached. The effect of these old records was that after the BTS was produced an incorrect figure was generated for the Opening Balance of the Local Suspense Account for the following period. This amount corresponded to the value of the historical record. These orphaned records were created between 16 th November 2010 and 9 th December 2010. When the next Trading Period was balanced, then this incorrect Opening Figure would result in the total value for Local Suspense being calculated incorrectly and the SPMR being asked to make good an incorrect amount . It is at this point that transactions would be generated into the audit trail reflecting the fact that the SPMR had cleared the Local Suspense account for an incorrect amount. The audit trail operated correctly in the sense that it accurately recorded the transaction on the system. This problem was not reported to Fujitsu in 2011/12 and only affected a small number of Branches and only for a single Trading Period. However the two branches with the largest discrepancies did report the issue to Post Office Ltd who could see the impact of the problem in their back end system and wrote off the loss or gain for the branch but did not ask Fujitsu to investigate further. At the same Trading Period in 2012/13, the problem re-occurred and this time one of the affect Branches reported the problem to Fujitsu on 25 th February 2013 (PEAK 223870) resulting in a detailed analysis of this issue and finding the orphaned BTS records. The root cause was determined by 28 th February 2013 and a preliminary report was sent to Post Office Ltd. A further update was sent on 14 th March 2013 with a full analysis of the issue and all the affected branches.”
“There possibly ought to be a different definition for it because equally data in the database which is telling you which stock unit, you know, what -- which TP [ie trading period] you're in, whether a stock unit is in use, all of those are operational data which do not go down and impact the transaction data, so for me, the definition of transaction data -- I should be able to get from one opening -- one set of opening figures to the end opening figures by going back to the transaction data in its absolutely raw form.” 452. He said it was what one went through to bring different totals together “to get to your end position in a trading statement.”
“We understand from Fujitsu that the SSC has been carrying out further work to identify any PEAKs that show transactions being injected at the counter in Legacy Horizon in addition to those referred to in paragraphs 29 and 30 of Mr Parker's second statement. On Monday [ie the 6 th day of the Horizon Issues trial] we learned that an SSC technician has:- • searched for all KELs that mentioned "RiposteMessageFile", "Ripostelmport" and "RiposteMessage'; • collated the responsive KEL references (AChambers2226M, CObeng1029162824, DSeddon822M, MYoung5043M, RColeman1250R, acha2340K, ballantj498J, dsed344J, AHolmes3343J, DSeddon1753N, GMaxwell46141, PCarroll12541, RKing5135L, 81111, pcar847S, wbra716s); • re-searched the Peak system for any PEAKs which contained those KEL references; and • identified the following PEAKs: PC0105560, PC0106885, PC0063599, PC0063871, PC0065796, PC0066061 and PC0083998 (which cross refers to PC0076029). We understand that the PEAKs referred to above relate to either:- • correcting configuration data after a PinPad change; or • reference data. We also learned that during this exercise the SSC technician has also identified three examples of the marooned transaction scenario described in paragraph 38.2 of Mr Parker's second statement: PC0068495, PC0099141 and PC0079196.” 467. The search terms identified in the letter were different to the ones that Mr Parker had used in his 2 nd witness statement. These were "RiposteMessageFile", "RiposteMessage", "LPO Delete", "Marooned" and "RiposteObject put". The reason this is important is the command “RiposteImport”, used by the SSC as a search term during the week commencing18 March 2019 , but not used for the purposes of Mr Parker’s 2 nd witness statement in January 2019, is actually one of the commands that was in fact used to inject transactions. In other words, the exercise carried out for the purposes of the witness statement was wholly deficient, in that it did not search for one of the specific terms used to inject transactions. 468. When asked to explain this in his cross-examination, and the corrections that he made, Mr Parker stated the following: “A I notified the legal team that Mr Simpkins had done some more work and as a result of that we changed -- we -- this letter was actually generated. Q And you didn't think it was important to correct paragraph 29 of your second witness statement accurately to reflect the situation as it would have been on the day you were due to give evidence, or indeed today? A That wasn't a choice I made personally . I was advised that we generated this letter.”
"As I explain below, those suggestions are incorrect and Mr Roll's account of Fujitsu's actions and powers is inaccurate and misleading". 471. He explained this paragraph as follows in his cross-examination. “A. In that paragraph I am trying to make the point that the suggestion that we frequently changed branch transaction data without informing the branches that such actions were being actually taken is not correct. "
“Q Now, given the change of your evidence in your later statements about the ability remotely to access and inject transaction data, can you explain to the court what you were saying you believed about that sentence? What should the court now read for those words in the light of the three statements that it has before it? A If I take the last sentence in isolation, which is what I think you are asking me to do, then I don't understand how I apply it, because I am -- I have been simply trying to say there that the frequency was not high, and that we would always involve the Subpostmaster wherever possible if that sort of action was actually being taken. That's what I'm trying to say by that.” 473. This was a very poor attempt by Mr Parker at explaining away quite directly incorrect factual evidence in a witness statement on a very important area in the litigation, namely Fujitsu’s ability to inject transactions remotely. Mr Parker’s only recourse was to claim he had not been saying what he plainly had been saying in his 1 st witness statement – that Fujitsu did not have the power to do it, and that Mr Roll’s evidence to the contrary was inaccurate and misleading. The power to do something cannot be equated with frequency. 474. Mr Parker also accepted that at the time he signed his 1 st witness statement he had known that Fujitsu had the power to insert transactions into individual branch counters by using the correspondence server, a process Mr Roll described as “to piggy back through the gateway”
“I now have pressing evidence to suggest that unwanted peripheral input is occurring, the likely source being the screen. This has been seen at Old Isleworth ...When the PM has been asked to leave the screen on overnight I have observed system activity corresponding to screen presses happening with no corresponding evidence of either routine system activity or human interference. The way forward now is to correlate this with Microtouch complied monitoring software and to this ends Wendy is arranging for installation of the kit on Friday..." However, notwithstanding this, the subsequent entries include “Phantom transactions have not been proven in circumstances which preclude user error" and “In all cases where these have occurred a user error related cause can be attributed to the phenomenon?”. 482. The use of the phrase “in all cases” in respect of attributing user error wholly ignores that the ROMEC engineer is said to have seen the problem themselves, and it is not simply the word of the SPM in this instance. It ignores the activity observed overnight by the Fujitsu employee who made the entry in the PEAK, when the branch was not even open. Also, there is no reason, given the lengthy entries and studying the text throughout the PEAK in detail, why user error should be reached as a conclusion at all. This appears to be been used simply as an excuse to avoid an adverse conclusion to Fujitsu. 483. Notwithstanding the detailed entries in the PEAK itself that in my judgment plainly point to the contrary, the PEAK was still categorised as No Fault in Product. 484. In another, dealing with phantom transactions, one branch was described as “a problem office” that was accordingly being “monitored”
“Q. Pausing there, if the PM is correctly reporting that, then that would be very serious for the Postmistress or Postmaster, wouldn't it? A. If it is being correctly reported, yes. Q And it would not be the system working as it should. A If we attribute it as a system fault, yes.” 491. This again was categorised by Fujitsu as Category C, non-critical. In a sense, those two questions and answers sum up a large part of the case. The reason that this plainly, in my judgment, does demonstrate a system fault is Fujitsu’s own PEAK recorded: “Voiced NBSC quoted ref H18174172 to see what checks they have done themselves before transferring call to Horizon. They stated they had trainers come into the office and ruled out user error .”
“I can't tell why the declared cash doesn't match the expected cash figure, the branch need to make sure that what they have recorded on the system is correct, and investigate the anomalies .”
“Ernst & Young have carried out the Horizon ISAE 3402 Audit since the 2012/2013 financial year (preparations for the audit began with Post Office in 2011)” 505. The second was a reference to how Ernst & Young performed the ISAE 3402 audit for the period1 April 2014 to31 December 2014 . His failure to mention the 2011 Management Letter is, in my judgment, a serious omission. 506. He could not be asked any questions about his evidence; although that was not his fault, this goes to the weight which I attach to his evidence in any event. 507. I consider that Mr Membury’s evidence is of limited, if any, assistance in resolving the Horizon Issues. It does however continue the very one-sided picture presented by all the Fujitsu witness statements, which was to omit any reference to important contemporaneous documents that criticise or demonstrate any deficiencies with Horizon. The absence of Mr Gareth Jenkins 508. It is entirely a decision of the parties which witnesses they choose to call in any proceedings in respect of any evidence. The position of one person, however, who did not appear in the Horizon Issues trial, must be considered in more detail than would be usual, as the claimants make considerable complaint about this. The person in question is Gareth Jenkins, a senior Fujitsu employee who, although he retired recently, was obviously widely available to the Post Office and the source of a great amount of information both to the Post Office’s witnesses of fact, and also to Dr Worden (although he was not separately identified in Dr Worden’s “sources of information” paragraphs in his 1 st Report). The fact that he provided information to Dr Worden emerged during the latter’s cross-examination. Mr Jenkins had previously given expert evidence for the Post Office in some of the criminal prosecutions of SPMs, in particular that of Ms Misra, to whom I have referred above, who was convicted of criminal offences in Guildford Crown Court in 2010. 509. When the Post Office served its evidence of fact, there was no witness statement from Mr Jenkins, although many of their witnesses relied upon him as their source of information, he was referred to very often, and he obviously knew a great deal about Horizon. The extent and way in which Mr Jenkins had been closely involved was explained by Mr Godeseth in his cross-examination. Mr Godeseth had, in respect of the receipts and payments mismatch, originally stated in paragraph 42 of his 2 nd witness statement that 60 branches were affected. He had corrected this to 62, a factual correction that was specifically made by him. The following passage of evidence is relevant to Mr Jenkins’ involvement in this. “A. No, no, I didn't do a calculation to come up with the 60. I was quoting from other people. Mr Justice Fraser: Someone just gave you the 60, did they? A. I thought I was quoting from other people. I -- Gareth even said to me that in my statement I had said "approximately 60", so I was not -- clearly I didn't because the statement here doesn't contain that word. I had rather hoped it had when this was first brought to my attention, but no, I certainly did not do any specific calculation to come up with the 60 that I put into my original statement. Mr Green: Did Gareth explain the change to you from 60 to 62? A. No. Q. But how did he come to tell you what was in your original statement? What was that conversation? A. I picked up the number -- to be -- my objective in this was to explain to the court the symptoms of the bug and how -- the technical aspect of it. I did not pay particular attention to getting the detail on how many branches were affected, correct. Q. Okay. So you have spoken to Gareth since your statement? A. I don't think I have spoken to him about this in particular. I was -- as I say, when Gareth had said I had originally said "approximately 60" I was thinking that was quite neat, but that's not the case. Q. Well, you said "Gareth even said to me that in my statement I had said 'approximately 60'", so he must have said that to you after your statement had been filed? A. It was a comment in a document that we were exchanging. Q. But you hadn't spoken to him about remote access since your first statement? A. No. Q. Why have you stayed off that topic with him? A. Oh, sorry, this was just a comment. We have been exchanging documents, we have been commenting on documents, so it was not a particular conversation. It is merely a case of Gareth had commented on this when it was pushed back to us that I had originally said 60 and actually the answer was different.” 510. There is nothing wrong with Mr Godeseth correcting the number of affected branches from 60 to 62. There is also nothing wrong with him reproducing a number given to him by someone else, as long as it was properly identified that the information came from someone else, who ought to be identified (which it wasn’t in his statement, but was, in at least outline terms, in his cross-examination). The reason for reproducing these passages is simply to identify the extent to which Mr Jenkins was so closely involved in the litigation, a point relied upon by the claimants. There were a great many references throughout Mr Godeseth’s evidence, written and oral, about information he had obtained from Mr Jenkins. 511. When the Post Office served their evidence of fact, the claimants had asked the Post Office why there was no statement from Jenkins, whether Mr Jenkins was available to give evidence, and also whether he was involved as one of a team of what the claimants referred to as the “shadow experts”
“144. [The claimants] understandably complain that Mr Jenkins and the other source of Mr Godeseth’s information could have given some of this evidence first hand. However: 144.1 Taking into account that Mr McLachlan’s evidence specifically addressed things said or done by Mr Jenkins in relation to the Misra trial, Post Office was concerned that the Horizon Issues trial could become an investigation of his role in this and other criminal cases. 144.2 Moreover, Post Office was conscious that if it only adduced first hand evidence in the trial, it would end up having to call more witnesses than could be accommodated within the trial timetable. 144.3 Furthermore, so far as Post Office was aware, the relevant parts of Godeseth 2 were most unlikely to be controversial. For example, the Misra trial was a matter of public record, the four bugs were covered by contemporaneous documentation and Post Office had no reason to doubt Fujitsu’s account of the documents it held.” 513. In a footnote to paragraph 144.2 of the closing submissions, the Post Office added “…..As noted above, had its witnesses only given first hand evidence, Post Office estimates that some 34 additional witnesses would have been required.” 514. The following are relevant in my judgment: 1. Of primary importance is the principle that it is for each party to decide whom to call as a witness, and what evidence they may seek to obtain from any particular witness. 2. Mr Jenkins is an important and central person so far as the operation, efficacy and robustness of Horizon is concerned, and also in respect of the number of incidents over the years that have led to PEAKs, KELs, problems and fixes. That is reinforced by the Post Office’s own closing submissions that stated, at paragraph 138, that there were two possible candidates for the witness giving the overview of Horizon, namely Mr Godeseth and Mr Jenkins. It also stated that taking into account Mr Jenkins’ involvement in cases before the CCRC (including the Misra case) the decision was taken to use Mr Godeseth. For that decision to have been taken, it is implicit that Mr Godeseth must know enough about the system to give the evidence that he did. 3. The point in the footnote about a potential further 34 witnesses is not relevant to whether Mr Jenkins was, or was not, called by the Post Office. The Post Office did not at any stage apply to the court and explain that further time was required for the length of the Horizon Issues trial for the specific reason that relevant evidence of fact from the Post Office would require a particular minimum number of witnesses. However, given the primacy of point (1) above, it was entirely for the Post Office to decide whether to call Mr Jenkins or not, how many witnesses to call, whom those witnesses were to be and what the content of their evidence should include. 4. The Post Office chose to proffer a reason for Mr Jenkins’ absence in closing submissions. They were not obliged to explain. However, the reason in paragraph 144.1 quoted above is not a valid reason for his absence. The claimants would have been entitled, in cross-examination, to put to Mr Jenkins any previous inconsistent statements he had made on the same subject, but obviously only if there were any. These could, potentially, have included previous statements he may have made in earlier proceedings, but in order to be allowed to do that, such statements would have had to be inconsistent with his evidence in the Horizon Issues trial. Putting a previously inconsistent statement on a particular fact is permitted as part of cross-examination if a witness’ evidence on that fact is in issue. However, the Horizon Issues trial would not have become “an investigation of his role in this and other criminal cases” – the Horizon trial was about the Horizon Issues. Also, this type of cross-examination would only have arisen if any statement(s) in the Horizon Issues trial was or were not consistent with those he had made in previous cases in which he had been involved. 5. Regardless of the validity of the explanation given by the Post Office, Mr Jenkins was not called as a witness, and it merits repetition that it is entirely a matter for the Post Office which witnesses it called, and which it did not. No speculation is permitted as to what evidence he might have given in the Horizon Issues trial. It ought to be recorded however that no limit on the number of witnesses was imposed by the court on any party, and the Post Office could (had it wished) have called both Mr Godeseth and Mr Jenkins. Nobody forced the Post Office to choose one or the other. 6. There is established authority that in certain circumstances the court can draw adverse inferences from the absence of a witness. The claimants do not invite me to draw any adverse inferences from the absence of Mr Jenkins, and I do not do so. 7. The fact that the Post Office chose to advance certain evidence in the Horizon trial, that emanated from Mr Jenkins, by means of another witness saying “Mr Jenkins told me that…..” means that the claimants were deprived of the opportunity properly to test that evidence by asking Mr Jenkins about it. It also emerged in cross-examination, although not in his written report, that Dr Worden has also obtained certain information directly from Mr Jenkins. By dealing with the material in this way, and having deprived the claimants of the opportunity to cross-examine Mr Jenkins, the weight to be given to evidence emanating from Mr Jenkins is less than it would be otherwise. 515. There are two consequences of the Post Office choosing to adduce such evidence in this way. The first is that Mr Jenkins did not have to undergo any cross-examination on such evidence; the second is that such evidence would not be given the same weight by the court as though Mr Jenkins had given it himself. The Post Office would have known this in advance, as this is wholly conventional. How these different factors were weighed by the Post Office and its legal advisers in deciding whether or not to call him, is solely a matter for the Post Office. The end result is that Mr Jenkins did not appear and was not cross-examined. 516. However, I reject the suggestion that Mr Godeseth was insufficiently knowledgeable of the matters upon which he was cross-examined for his evidence to be of significant evidential weight. The Post Office’s Closing Submissions sought to dilute the effect of Mr Godeseth’s cross-examination by, at the same time, implicitly suggesting that Mr Jenkins (or someone else, but not Mr Godeseth) would have been a far better person to have answered those questions; yet explaining why Mr Jenkins had not been called. These submissions, though understandable from the Post Office’s point of view in forensic terms only, have little force. A clear choice was made by the Post Office not to call Mr Jenkins as a witness; that was their right. A clear choice was also made to call Mr Godeseth. The Post Office was are entitled to call such evidence as deemed necessary by both the Post Office and its advisers. They cannot, however, have matters both ways, and try to downplay Mr Godeseth’s evidence because he was called to deal with certain matters, and Mr Jenkins was not. The Post Office has to abide by the consequences of their choice of witnesses, in terms of the evidence now before the court to resolve the Horizon Issues trial. I have already explained my view of Mr Godeseth’s evidence in Part E above, and it need not be repeated. Inaccurate Statements by the Post Office 517. Litigation in this jurisdiction is adversarial. What that means is each side advances their own case, and challenges or rebuts the case of the other side, and the court assesses the evidence before it adduced by the parties, makes necessary findings on the facts and applies the law. Parties in civil litigation will usually advance their own evidence of fact, and although mechanisms are available for compelling other witnesses to attend and answer questions, there can be some pitfalls in doing so and issuing a witness summons (what used to be called a subpoena) is a relatively rare step. The court can only therefore resolve the case on the evidence before it, although it can draw inferences, that is to say common sense conclusions, on the evidence it does have. 518. The Post Office has no obligation to assist the claimants in advancing their case against it. The Post Office has, however, maintained publicly that it was seeking to be “transparent” about Horizon, and prior to the litigation it made certain public statements in relation to the increasing disquiet on the part both of SPMs, and others who became involved either on their behalf (such as some Members of Parliament) or in an investigative way (such as the BBC Panorama programme and other journalists). These statements by the Post Office routinely and strongly insisted that there was nothing in the criticisms being levelled at the accuracy of Horizon, and that losses that were shown in SPMs’ branch accounts were caused either by carelessness or dishonesty on the part of the different SPMs who experienced what they considered to be unexplained discrepancies and losses. In 2015 a Parliamentary Select Committee held hearings into the mediation scheme that had been set up to attempt to address the claims by SPMs in respect of Horizon. I have not considered any evidence submitted to the Select Committee, or any details of the mediation scheme (which was being conducted by Second Sight) and refer to them here simply as a matter of chronological record. 519. The claimants rely upon a public statement released by the Post Office in 2015 after the BCC Panorama programme. Part of the statement was headed “The Horizon System” and stated: “The Horizon Computer System Horizon is robust and effective in dealing with the six million transactions put through the system every day by our postmasters and employees at 11,500 Post Office branches. It is independently audited and meets or exceeds industry accreditations. There have been 500,000 users of the system since it was introduced. Nevertheless, rigorous re-investigations were undertaken into claims made by 136 mainly former postmasters that the system caused losses in their branches. There is overwhelming evidence that the losses complained of were caused by user actions , including in some cases deliberate dishonest conduct. The investigations have not identified any transaction caused by a technical fault in Horizon which resulted in a postmaster wrongly being held responsible for a loss of money. There is also no evidence of transactions recorded by branches being altered through ‘remote access’ to the system. Transactions as they are recorded by branches cannot be edited and the Panorama programme did not show anything that contradicts this.”
“Dear both, your help please in answers and in phrasing those answers, in prep for the SC: 1) "is it possible to access the system remotely? We are told it is." What is the true answer? I hope it is that we know this is not possible and that we are able to explain why that is. I need to say no it is not possible and that we are sure of this because of xxx and that we know this because we have had the system assured.”
“As discussed, can you hook up with Kevin to review what answers have already been provided to second sight as this should form the Post Office response.” 522. The answer was provided by Mark Underwood on30 January 2015 in an email which is part of the same email chain or string: “ Can Post Office or Fujitsu edit transaction data without the knowledge of a Subpostmaster?”
“Mark / Mel, James has had a look at your answer to Q1. And thinks there’s too much detail for Paula – this was written for a different type of audience. He has captured the same points but in a more appropriate format :- He states:- Having looked again at the request from Paula, it appears that the fundamentals around this question (remote access) are not understood . I suggest that Paula is briefed along the lines of the following. 1) No transaction data is held locally in any branch. Transactions are completed and stored in a central database and copies of all data is sent to a secure audit database. 2) Sub-postmasters directly manage user access and password setting locally so system access (to create transactions) are limited to approved local personnel only who are responsible for setting their own passwords. Users are only created following an approval process which requires authorisation by the sub-postmaster. All subsequent transactions are recorded against the id used to log on to the system. 3) Once a transaction has been completed, there is no functionality (by design) for transactions to be edited or amended. Each transaction is given a unique number and ‘wrapped’ in a digital encryption seal to protect its integrity. All transactions are then posted to a secure and segregated audit server. 4) On approval, there is the functionality to add additional transactions which will be visible and have a unique identifier in the audit trail. This is extremely rare and only been used once since go live of the system in 2010 (March 2010) 5) Support staff have the ability to review event logs and monitor, in real time, the availability of the system infrastructure as part of standard service management processes. 6) Overall system access is tightly controlled via industry standard ‘role based access’ protocols and assured independently in annual audits for ISO 27001, Ernst and Young for IAS 3402 and as part of PCI audits.”
“25. Further, the Defendant was, by itself and/or via its agent Fujitsu, able to alter branch transaction data directly and carry out changes to Horizon and/or transaction data which could affect branch accounts. 26. However, the Defendant has made public statements in the following terms: 26.1. "Horizon does not have functionality that allows Post Office or Fujitsu to edit or delete the transactions as recorded by branches" (Defendant's published reply to Second Sight's Briefing Report Part Two, concerning a review of the Horizon system); and 26.2. "Transactions as they are recorded by branches cannot be edited" (Defendant's published reply to the BBC Panorama documentary in relation to Horizon). 27. These statements were untrue, as the Defendant now admits.” 531. These paragraphs were pleaded to by the Post Office in its Generic Defence in the following terms: “57. Paragraph 25 appears to be concerned with the editing or deletion of transaction data input by or on behalf of a Subpostmaster without his or her consent. Accordingly, Post Office assumes that it is not concerned with transactions such as Transaction Corrections which are sent to branches but must be accepted by or on behalf of the Subpostmaster before forming part of his or her branch account. As to the circumstances in which such transaction data can be edited or deleted without the consent of the Subpostmaster: (1) Neither Post Office nor Fujitsu has the ability to log on remotely to a Horizon terminal in a branch so as to conduct transactions. (2) A Post Office employee with "global user" authorisation can, when physically present at a branch, use a terminal within the branch to add a transaction into the branch's accounts. The purpose of "Global User" authorization is to allow access to the systems for during training and/ or audits. Any transactions effected by a Global User are recorded against a Global User ID and are readily identifiable as such. (3) Fujitsu (and not Post Office) has the ability to inject transactions into branch accounts (since the introduction of Horizon Online in 2010, transactions of this sort have been called "Balancing Transactions"). These transactions do not involve any removal or amendment of the transactions entered at the branch. Their intended purpose is to allow Fujitsu to correct errors or bugs in Horizon by cancelling the effect of an error or bug on a branch's data. They may be conducted only by a small number of specialists at Fujitsu and only in accordance with specific authorisation requirements. They are rarely used. To the best of Post Office's information and belief, only one Balancing Transaction has ever been made so as to affect a branch's transaction data, and this was not in a branch operated by a Claimant . A Balancing Transaction is readily identifiable as such. (4) There are a small number of Fujitsu specialists who have certain privileged user access rights which they could in theory use to amend or delete the transaction data for a branch. The intended purpose of privileged user rights is system support, not the alteration of branch transaction data. To have abused those rights so as to alter branch transaction data and conceal that this has happened would be an extraordinarily difficult thing to do, involving complex steps (including the writing of sophisticated computer programmes and circumvention of sophisticated control measures) which would require months of planning and an exceptional level of technical expertise. Post Office has never consented to the use of privileged user rights to alter branch data and, to the best of its information and belief, these rights have never been used for this purpose. (5) Post Office cannot conceive of a reason why any Fujitsu personnel would have sought to add, inject, amend or delete any transactions in any branch accounts so as to create a false shortfall. It would for all practical purposes be impossible for any of them to generate significant shortfalls without detection and, even if they were able to do so, they would be unable to take the benefit of such shortfalls for themselves.”
“When I talk about remote access I'm talking about action taken remotely to either inject new transactions or to edit existing transactions or to delete existing transactions in a way that could change the accounting position of the relevant branch.”
“Master, first of all, could I just deal with the remote access point? The letter to which my learned friend took you was, as you might expect, written by people who thought it was correct. The Horizon system is a very complicated system. It involves lots of departments in ... both in Fujitsu and in the Post Office. And the people who are responsible for the correspondence didn't know that, in fact, there were these two other routes. Very few people at Post Office knew that there were these two other routes. They were ... they were routes that are under ... essentially under the control of Fujitsu who's the expert independent contractor that is involved in the operation of the system . And it is a matter of enormous regret that the people who wrote that correspondence and made those submissions weren't aware of that but, you know, we are where we are; the point is that, the point having been discovered, the Post Office wasted no time in ... in bringing the truth ... the accurate ... and accurate set of facts to the knowledge of the claimants.”
“Taking remote access as an example, the need for remote intervention affecting branch accounts will obviously be rare. On any view, the occasions on which privileged users at Fujitsu have exercised their ability to remotely inject, edit or delete branch transactions or accounting entries will represent a tiny percentage of the relevant transactions/accounting entries. And the occasions on which they have done so negligently or dishonestly will, in turn, represent a very small percentage of those occasions . So, compared with the volume of business recorded in branch accounts, the number of cases in which false data will have been remotely introduced will be extremely small (multiplying a small chance by a small chance). This is a “second order effect” (a small proportion of a small proportion) which is, by definition, extremely unlikely to have any significant impact on the robustness of Horizon.”
“Q. Dr Worden, you knew that a central issue, not only a central issue legally but a very high-profile issue in the case, was the extent to which Post Office had remote access to the counters, didn't you? You knew that? A. Yes, and what I'm talking about, what I was talking about was the extent to which this could happen without the knowledge of the subpostmaster. Q. And that's the – A. And we agreed in the joint statement that more or less Fujitsu or Post Office could do anything.” 541. He was asked about this answer two days later. “Q. It is Day 18, page 67. I think it is lines 5 and 6 actually. It says: "Answer: And we agreed in the joint statement that more or less Fujitsu or Post Office could do anything." A. Yes. We can see precisely what we agreed in a minute, obviously. Q. In relation to the – A. We agreed the experts couldn't demonstrate that they couldn't do everything. I mean, that's sloppy wording by me there. I think the joint expert statement says it better.” 542. The exact wording in the joint statement in respect of repairing transactions is as follows at 10.2 of the 4 th Joint Statement: “Certain facilities and procedures used by Fujitsu to repair the more common issues which arose in Horizon were standardised, and evidence of them persists. However, to repair less common issues which arose from time to time, standard tools and procedures might not have been sufficient, and evidence might not persist of what was done at the time. Even when evidence does persist, it may be extremely difficult for the experts to interpret it today, because of the scale and complexity of Horizon. Therefore, it is usually difficult for the experts to make categorical negative statements of the form: ‘X or Y never happened’.” 543. This litigation is between the claimants and the Post Office. Fujitsu is not a party. Although one of the contracts between the Post Office and Fujitsu was uploaded to the electronic trial bundle on the penultimate day of evidence (by the Post Office’s solicitors) the litigation is not concerned with the details of that contractual relationship, other than (tangentially only) the cost to the Post Office of making ARQ requests, as the ARQ data is maintained by Fujitsu and charges are raised to the Post Office in this respect. However, Fujitsu’s involvement does cast something of a shadow. That contract has been addressed in terms of whether certain documents were, or were not, in the control of the Post Office. I have already dealt with the issue of PEAKs and KELs, and how the Post Office maintained initially it was not obliged to disclose these. Fujitsu personnel who were not witnesses were involved in providing information to the Post Office’s expert, Dr Worden. It was the Fujitsu witnesses who originally, and in my judgment unequivocally and directly, flatly contradicted what Mr Roll said could be done on the subject of remote access. I have also identified that in my judgment, the majority of the Fujitsu witnesses were more interested in following some sort of Fujitsu party line than they were in answering questions in cross-examination wholly frankly, although I exempt Mr Godeseth from that criticism. From what was said by the Post Office’s leading counsel to Master Fontaine at the very first case management hearing, those within the Post Office had relied upon Fujitsu in obtaining relevant information for its earlier public statements, and it plainly relied upon Fujitsu in terms of evidence of fact in the Horizon Issues trial, because it called a number of Fujitsu witnesses. I am unaware of the degree to which the Post Office has depended upon Fujitsu in the background to the Horizon Issues trial, and there is no reason why I should be, and I make it clear that I do not speculate on this. This information must have originated from Fujitsu at some point, but the Post Office must bear some responsibility for such incorrect statements having been made before, both publicly and in its pleadings. 544. Nor could it be said, in the Post Office’s and/or Fujitsu’s defence, that remote access was a side issue. It has been a very important central element of the whole dispute between SPMs and the Post Office over the Horizon system for many years. Mr Green for the claimants adopted a non-computing analogy, namely that of “a back door”
“What is the true answer?” 546. She also said in the same email “I hope it is that we know this is not possible and that we are able to explain why that is”
“In paragraph 20 of Roll 2, Mr Roll describes a process by which transactions could be inserted via individual branch counters by using the correspondence server to piggy back through the gateway. He has not previously made this point clear. Now that he has, following a discussion with colleagues who performed such actions I can confirm that this was possible. I did not mention it in my first witness statement because, when faced with a less clear account in Mr Roll's first statement, my recollection was that if it was necessary for the SSC to inject a transaction data into a branch's accounts, it would have been injected into the correspondence server (injecting via the server was the default option which was followed in the vast majority of cases).” 553. The recusal application was issued the day after Mr Godeseth’s cross-examination had made it clear, not only that this remote access existed, but after he was taken in careful cross-examination through specific examples of Fujitsu personnel manipulating branch accounts, and leading to discrepancies in branch accounts. I am aware that criticism of the Post Office and Fujitsu in this respect may prove to be controversial, as earlier criticism of certain aspects of the Post Office’s case in Judgment (No.3) was not well received by it. However, if criticism is justified, I consider it would be detrimental to proper resolution of the group litigation if that criticism were to be withheld simply because it might lead to a further negative reaction by the Post Office. It is also an inherent part of the judicial function in any litigation to make findings, which may include criticisms where justified, that may be contrary to a litigant’s own view of the merits of their case. Some litigants are so convinced of the righteousness of their own position that they consistently refuse to accept any possible view of the litigation other than their own. Such a blinkered view is rarely helpful, and would be particularly unhelpful from a publicly owned institution. 554. I consider that criticism is justified of the Post Office’s incorrect previous statements – which included public and high profile statements such as the response to the BBC Panorama programme, whatever may have been said on the subject to the Select Committee, as well as in its own pleadings - about remote access without the knowledge of SPMs. The Post Office should have done its best to discern whether such remote access was possible when this subject first arose, and whether it had occurred, before these statements were made. Fujitsu should have been frank and unequivocal, internally, with the Post Office, so that there could be no possibility of incorrect statements on this important point being made publicly by the Post Office. The Fujitsu witnesses should not, in their first witness statements, have made the incorrect statements that they did. Had those initial statements been factually accurate, there would have been no need for the supplementary statements from them that eventually led to the true position being accepted by Fujitsu, and therefore by the Post Office. It is highly regrettable that such a situation as this should have developed. I am not making any findings on any future issues, whether of fact or law, yet to be tried. 555. There is however considerably more to resolving the Horizon Issues than simply making findings about remote access to branch accounts. So-called “shadow experts” instructed by the Post Office 556. I return to the issue of so-called “shadow experts.”
“The Defendant has spent around£500,000 on investigations by internally appointed experts for the purposes of determining its litigation strategy. The resulting material - which is privileged - has not been provided to the Defendant's expert for the purposes of this litigation.” 557. The claimants adopted the term “shadow experts” for these, a term with which the Post Office did not agree. At the cost management hearings, the claimants pointed out that by “internally instructed” this meant these experts were not instructed by the Post Office’s solicitors. It was also pointed out that there was no corresponding entry in the budget for any conference with any of their counsel, and this meant that this was “an entirely ring-fenced operation”
“I6. The Court’s interventions on disclosure 1145. Post Office is concerned that the Court may have lost sight of the nature of the Orders made and of the approach which the Court ordered. It is striking that Cs have not themselves made any applications for specific disclosure nor have they advanced any complaint that particular disclosure orders have not been complied with. 1146. The recent judgment of the Court of Appeal in Serafin v Malkiewicz & ors[2019] EWCA Civ 852 @ para 118 is relevant: “We are also highly troubled by the repeated demands and criticisms by the Judge regarding the Claimant’s disclosure, in circumstances where pre-trial disclosure had been completed by both sides at a time when both the Claimant and Defendants had been represented by solicitors and counsel, and no application for further disclosure had been made by the Defendants…” ” 563. These submissions identified what were said to have been “the Court’s interventions on disclosure” in the Horizon Issues trial. The authority identified by the Post Office, and stated to be relevant to that submission, concerned a defamation case in which the judge had demanded disclosure of a litigant in person, and made criticisms of him, both during the trial of that litigant’s libel claim, and in the judgment itself. In some instances the court in that case had, of its own volition, demanded production of certain documents from the litigant in person, and within a very short timescale. This had been part of findings made by the Court of Appeal in respect of procedural irregularity and the successful appeal against the judgment. 564. There were no “interventions on disclosure” by the court during the Horizon Issues trial, nor were there “demands and criticisms”
“KELs and PEAKs together form a useful source of information about bugs in Horizon but are a limited window on what happened. It is sometimes necessary to use evidence from both to try to understand, but even so they are not a comprehensive picture. It is to be expected that both KELs and PEAKs are incomplete in various respects.”
“We understand that Fujitsu maintained a 'Known Error Log' for Horizon and that such reports will have been provided to Post Office. Please see the list of the categories of documents relating to Fujitsu referred to below, that we request disclosure of.”
“In circumstances where you have not particularised any factual basis on which Horizon is defective, disclosure of these documents (if they exist) is not relevant, reasonable or proportionate.” 577. The suggestion in that letter that the Known Error Log was not relevant, is simply wrong, and in my judgment, entirely without any rational basis. The further suggestion, viewed with the hindsight now available, that the “known error log” may not exist, is disturbing. The claimants’ request used the precise title – “known error log” – and this clearly did exist. To suggest in an answer “if they exist” is somewhat misleading. 578. Item 23 in the same list of documents sought was “Internal memoranda from Fujitsu and POL referred to by Second Sight as identifying a `Horizon bug' within Horizon Online.” 579. The answer against that item was: “We do not recognise the document to which you refer. Please provide further details.” 580. In my judgment, the documents sought in that entry must clearly include any PEAKs that identified a bug within Horizon Online. “Internal memoranda” is a plural reference, yet it was interpreted by the Post Office’s solicitors as though it were singular, and the request was for a single document, or a document with the title “internal memoranda”
“The claims which you have particularised concern errors with the Core Audit Log. Following a review of the Known Error Log, Fujitsu have confirmed that there have been no logs in respect of Core Audit Log. The remainder of the Known Error Log does not relate to the claim which you have particularised and as such disclosure of this document is not relevant .”
“ It is admitted that Fujitsu maintain a "Known Error Log" . This is not used by Post Office and nor is it in Post Office's control . To the best of Post Office's information and belief, the Known Error Log is a knowledge base document used by Fujitsu which explains how to deal with, or work around, minor issues that can sometimes arise in Horizon for which ( often because of their triviality ) system-wide fixes have not been developed and implemented. It is not a record of software coding errors or bugs for which system-wide fixes have been developed and implemented. To the best of Post Office's knowledge and belief, there is no issue in the Known Error Log that could affect the accuracy of a branch's accounts or the secure transmission and storage of transaction data.”
“Cs’ response to the criticism of their case on Horizon is to argue that they do not have the access to information and documents that would allow them to plead any properly articulated case as to the “bugs” that they wish to blame for the shortfalls in their branches. Without prejudice to its position that the case is not properly pleadable, even at a generic level, Post Office is prepared to take steps to help Cs investigate Horizon, take an informed view as to whether they really wish to maintain their claims and, if so, decide how to proceed with them. Post Office wishes to put an end to the speculative attacks on Horizon and the related allegations as to its own conduct . It puts forward its proposals in paragraphs 2 to 8 of the Draft Order as a pragmatic compromise between the parties’ competing interests and concerns.” (italics present in original, emphasis by underlining added) 585. At the first case management hearing itself, upon specific enquiry by the court, this position was further maintained. 586. Disclosure of the Known Error Log was resisted by the Post Office on the grounds both of control, and content. So far as control is concerned, Mr Parsons had provided his 4 th witness statement for that hearing, the first before me as the managing judge, which expressly stated at paragraph 35 “ Despite Post Office explaining the irrelevance of the Known Error Log and that it was not within Post Office’s control…” and also at paragraph 37 “Due to the large amount of information involved and the fact that the Known Error Log is not in Post Office's control….”
“It will be clear from what I've just read to your Lordship that Post Office thinks that this is a complete red herring.” 588. The position was then, again, expressly confirmed as follows. “Mr Justice Fraser: I think it's going to be useful for me, certainly, for you to define all the different grounds upon which you currently resist disclosure of the known error log. Number (1) is you say it's not in your control. A: Control. Mr Justice Fraser: Number (2) you say the subject matter is not relevant. Is that right? A: My Lord, yes. An order that Post Office disclose documents relating to bugs and errors causing the branch account errors, if Post Office had control of the document and it dutifully complied with that order it would not be disclosing the known error log because, as far as it is aware, the known error log is not that kind of document.” 589. So far as the content of the Known Error Log is concerned, when asked by the court “does it contain any errors at all?” the following answer was given by the Post Office’s leading counsel: “It contains things like there's a problem with printers. There's a printer. You have to kick it on the left-hand side to make the printer work. I mean there's a vast range of hardware problems of that sort and maybe some software problems …. but not the kind of bugs, errors and defects that the claimants are wishing to pursue in their particulars of claim so far as Post Office is aware.” 590. This exchange with the court then continued. “Mr Justice Fraser: Well, that's the rider which slightly might concern--- A: Well, my Lord, unfortunately, that's all that Post Office can say because it's not Post Office's document. It's Fujitsu's document. Fujitsu are the experts.” 591. Given there was no witness statement from anyone at Fujitsu available on the subject at that hearing, that was as far as the matter could be taken on that occasion in terms of making any specific order relating to the Known Error Log. The Post Office had made clear that any order for disclosure that did not specify the Known Error Log, but was more general requiring disclosure of documents in relation to bugs and errors causing branch account errors, would not lead to disclosure of the Known Error Log in any event. A practical way forward, without requiring a fully argued separate application which would have included Fujitsu, was adopted whereby the IT experts were permitted to inspect the Known Error Log. The status of the Known Error Log as a disclosable document was resisted both on the grounds of control and relevance. It has to be acknowledged that this practical solution was one suggested by Mr de Garr Robinson. 592. It must therefore have come as a surprise to the Post Office, given the contents of its pleadings and the express submissions that it had made to the court, that both experts considered the Known Error Log to be highly relevant. This should also have led to the Post Office beginning to doubt what it was being told by Fujitsu, given the source of what the court was told about this was what Fujitsu had told the Post Office, as set out above. The explanation of what the Known Error Log was, what it contained, and its lack of relevance, was not remotely accurate. 593. Although a certain number of entries in the Known Error Log, which led to a significant number of different entries which are each called KELs being deployed in the trial and examined closely, were thereafter available, the issue of control of the Known Error Log and the Post Office’s earlier position resisting disclosure of this, did not go away. An entire appendix to the claimants’ opening submissions was devoted to criticism of the Post Office on disclosure, including its shifting position on the Known Error Log. In closing submissions, at the very end of the final day of the Horizon Issues trial, the claimants handed up a document entitled “Claimants’ Points in Reply”
“( 1) A party’s duty to disclose documents is limited to documents which are or have been in his control. (2) For this purpose a party has or has had a document in his control if – (a) it is or was in his physical possession; (b) he has or has had a right to possession of it; or (c) he has or has had a right to inspect or take copies of it.” 599. Sub-paragraphs (a) to (c) are not exhaustive, but collectively those two sub-paragraphs mean, in this case concerning this document or documents, that if the Post Office has, or had, a right to possession of the Known Error Log, and/or a right to inspect or take or be provided with copies of it, it is within the Post Office’s control. Control will be established if a party has a contractual right to inspect or take copies of the document or documents in question. Given the terms of the Fujitsu contract, the claimants submitted that the Known Error Log is in the Post Office’s control and has been at all material times. 600. The Post Office’s written submissions on this were also received. These conceded, as it was put in paragraph 3, that the KEL was in its control. That paragraph stated at sub-paragraph (1): “As at July 2017, Post Office understood the KEL to be a relatively unimportant internal working document produced and used by Fujitsu to assist in the performance of some services under the contract. On that basis, the KEL would not properly have been characterised as a “record…relating to the performance” of Fujitsu’s services under the Fujitsu Contract and would thus not have been within Post Office’s control. But, on the facts now known to Post Office, it would not contend the KEL to be outside its control.”
“25.8 In addition to its obligations under Clauses 25.2 and 25.3, Fujitsu Services shall provide the Court Case Support Services to Post Office in relation to prosecutions and other disputes (whether civil or criminal) with any third party including but not limited to any fraud, theft, breach of contract or impropriety (the “Court Case Support Services”). The Court Case Support Services shall include any matters whether they relate to Horizon, HNG-X or any other system provided by or on behalf of Fujitsu Services to Post Office , its agents or its subcontractors (including Post Office Service Integrator and any Tower Contractor). Fujitsu Services shall provide the Court Case Support Services within the timeframes required by Post Office or the relevant court or other authority. 25.9 Without prejudice to Clause 25.3, the Court Case Support Services shall comprise: 25.9.1 the provision of copy reports; 25.9.2 the provision of data (including transaction data, event logs, helpdesk call logs, non-polled data and remuneration data) where such data is held by or in the control of Fujitsu Services ; 25.9.3 the compilation of data (including transaction data, event logs, helpdesk call logs, non-polled data and remuneration data) ; 25.9.4 the interpretation of data (including transaction data, event logs, helpdesk call logs, non-polled data and remuneration data); 25.9.5 the provision of technical reports regarding technical aspects of any system (whether Horizon, HNG-X or otherwise ); 25.9.6 live witness evidence at Court if any of the information provided (including without limitation that provided pursuant to Clauses 25.9.1 to 25.9.5) is challenged to the extent to which Fujitsu Services provided said information; and 25.9.7 the right of access to Records, including but not limited to information, reports and data, held by or in the control of Fujitsu Services , and the assistance of Fujitsu Services’ personnel with appropriate knowledge of the applicable Records (to the extent any such personnel remain employed or contracted to Fujitsu Services) for any independent experts and/or legal advisors instructed by Post Office and/or any other claimant(s) or defendant(s) and the Prosecution in any mediation, arbitration tribunal, court case or dispute in which Post Office is involved in relation to the Horizon and HNG-X or any other system provided by or on behalf of Fujitsu Services to Post Office.”
“Records: means the full and accurate records relating to the performance of the Services.”
“ 4.5.1 Maintaining the Known Error Log on the SSC intranet site The SSC generates and maintains a Known Error Log (KEL) system that uses searchable documents in HTML format. The mechanism for searching is a query entry in an intranet site. The KEL system is available to first, second, third and fourth line support units as well as SSC staff. 4.5.2 Transferring knowledge between support units The SSC intranet site has KEL search facilities and other useful diagnostic data, documents and tools. SSC and SMC staff raise KELs based on customer-observed symptoms. KELs are further maintained once the fault has been resolved.” “4.7.1 Known Error Logs (KELs) The intranet site holds known error details in Microsoft Word format, the contents of which may be searched for, in full text form. Documents are created to a defined template wherever possible. An application has been generated which limits the properties of the document to a subset of possible values, for clarity and ease of search. This application is made available to all support units. The process for creating KEL entries outside of the SSC has not yet been formulated, but it is expected that no KEL will be allowed onto the system before it has been authorised by SSC staff.” (bold present in original) 607. The Post Office plainly did not know this when making the submissions that were made in 2017. The Services Operation Manual had been in existence for 16 years by then, and this ignorance about the Known Error Log, which is clearly explained within that document, is most surprising. 608. Even without such an express obligation in the Fujitsu contract itself, I find it very difficult to believe that Fujitsu could mount any sort of coherent or legitimate objection to producing it at the request of the Post Office. The assertion that the Log was not in the control of the Post Office is, in my judgment, not a valid one. 609. Similarly unsustainable is the submission that was made to the court in 2017 that its contents were not relevant to the Horizon Issues, and that it did not contain references to bugs and errors having an impact on branch accounts. On that latter point, the Post Office’s submissions were based upon what the Post Office had been told by Fujitsu. Fujitsu were, again, being clearly inaccurate about what they had told the Post Office was actually contained in the Known Error Log. 610. Nor is the passage quoted above in the Generic Defence an accurate description of the Known Error Log. The Known Error Log does far more than “explain how to deal with, or work around, minor issues that can sometimes arise in Horizon for which (often because of their triviality) system-wide fixes have not been developed and implemented”
“20. In the course of investigations into the bugs referred to by Mr Coyne and/or Dr Worden, including in the Second Joint Statement, on29 March 2019 Mr Lenton identified PCO273234 to Ms Simmonds as being a document relating to the Drop and Go Bug. For the next couple of months my firm continued to investigate these bugs and a number of further documents relating to these bugs came to light. PCO273234, along with 17 other documents (excluding redundant image files) that had come to my firm's attention when preparing for the Horizon Issues trial were disclosed in a single batch on31 May 2019 .”
“We write further to the email sent by the Claimants' solicitors to the Managing Judge's clerk on27 September 2019 at 12:29, which brought to the Managing Judge's attention that 32 documents relating to the Horizon Issues Trial were disclosed by Post Office on25 September 2019 . During the course of responding to a number of queries that the Claimants' solicitors had raised in relation to the disclosure of these documents (which can be located at [H/358] of the Horizon Issues Trial Bundle), it has come to Post Office's attention that a statement within its EDQ is incorrect and that there may be previous versions of the Known Error Log that have not been disclosed to the Claimants . Please find enclosed a letter which was sent to the Claimants' solicitors today explaining the background to these documents coming to light and the proposals which have been made in respect of providing any disclosure which may be sought by the Claimants.”
“In respect of your query regarding the fact that intervening changes to KELs are not captured on the face of the KELs disclosed, we have made further enquires with Fujitsu to confirm our understanding that previous versions of the KELs are no longer held. We regret to say that these enquiries have revealed that our understanding was wrong. As you will be aware, Post Office's EDQ stated that "[t]he KEL only contains the current database entries and is constantly updated and so the current version will not necessarily reflect the version that was in place at the relevant time. The previous entries /versions of the current entries are no longer available". This statement was based on info provided by Fujitsu. In response to our recent enquiries, however, we were informed by Fujitsu on30 September 2019 that this is incorrect and that previous versions of KELs are available. This takes Post Office greatly by surprise. It relied on the information provided by Fujitsu at the time of filing its EDQ that such documents were not available. It is extremely sorry that this information was incorrect. Regardless of whether these documents may or may not be "adverse", Post Office has taken immediate steps towards arranging for the previous versions of these KELs to be extracted and has instructed Fujitsu to begin producing a script to extract the documents into a readable HTML format. Unless you disagree, once this script has been produced we will ask Fujitsu to extract all of these versions. When this has been done, again unless you disagree, we propose to take immediate steps to disclose previous versions of the KELs which were referred to by either Dr Worden or Mr Coyne in their expert reports or the joint statements or the bug table, or were for any other reason included in the trial bundle. Disclosure of the previous versions of these KELs (where there are previous versions) will be given as a matter of urgency. We invite you to tell us whether the Claimants also wish to be provided with disclosure of previous versions of the KELs which were not referred to by the experts or included in the trial bundle. Given the seriousness of this matter, we propose to notify the Managing Judge of this development immediately and will send him a copy of this letter under cover of an explanatory email.”
" Q. First question about that: by amendment to your statement you now say that this was in around January 2018 rather than March 2018. A. Correct. Q. What caused you to make that correction? A. I had a look at the -- we hold the records for the information in the office, so I had my assistants look at the records, transaction logs and that's when I confirmed that it was January rather than March. (Inaudible) was logs in March as well. We made calls effectively every month to Horizon help desk concerning this issue. Q. When do you say you asked your assistants to check about the date? A. Yes. Q. Sorry, when do you say that happened? A. It was after I made the initial statement, I was checking. Q. Roughly when, Mr Latif? A. It would have been a few weeks ago, sir. Q. So is it right that you didn't check those records from the branch before making your witness statement? A. No, I thought I was correct but I double checked and made sure that actually in fact they were correct, those (inaudible), so I was right but initial incident happened in January when TA (inaudible) transaction acknowledgement, the TC, the corrections, they came in March. Q. Okay. So you say, do you, that the TCs in relation to these transaction acknowledgements came in March 2018? A. Yes. Q. And you say that's something you have checked from your records? A. Yes." 642. The Post Office made requests for disclosure during the trial, on15 March 2019 (which was the end of the first hearing week) requesting disclosure of various documents relating to Mr Latif and Mr Tank and their answers in cross-examination. This also included requests of CCTV recordings which Mr Tank said he had looked at. These requests were refused on27 March 2019 , and the claimants stated that this refusal was on the basis that they were either outside their control, no longer existed or were not relevant to the claimants’ evidence and were not relied upon at trial. 643. Documents that were held in Mr Latif's former branch were plainly outside of his control, even though his former branch manager had accessed them previously. However, that does not mean that the former branch manager could not at least have been asked for them. Neither the claimants did that, nor indeed did the Post Office (so far as I can tell). I consider the claimants’ attitude in this respect unhelpful, although it should be noted that for at least the documents in respect of Mr Latif, the Post Office could have taken its own steps in this regard. 644. There are some unsatisfactory aspects to the Post Office’s submissions about the CCTV footage. Mr Latif clearly said in his first witness statement at paragraph 7 “I also checked the CCTV to make sure that I had performed the transaction correctly.” Secondly, the Post Office’s closing submissions refer to requests for “the CCTV recordings which were consulted by Mr Tank for the purposes of drafting his witness statement”. He was asked about the CCTV during his evidence and did not say that he had consulted it for the drafting of his witness statement, he said that he had checked it at the time. Had the Post Office wished, it could have asked for that footage shortly after receiving that statement. It is also the case that the event in respect of which he referred to the CCTV occurred in July 2015. It may be that the CCTV images do not still survive, so long after the event in question, but this point was not pursued in cross-examination. 645. There is, however, one major difference between the claimants and the Post Office in this respect. The Post Office were entitled to, and did, explore with specific witnesses certain issues in respect of disclosure by asking the relevant person (in the above example, Mr Latif) questions about it. The claimants did not have this opportunity for the following reason. The majority of complaints made about the Post Office’s disclosure arise as a result of Fujitsu. Save for rare circumstances, such as Mr Parson’s specific witness statement about the Royal Mail issue, all that the claimants and the court are usually told is “Fujitsu say X” or “Fujitsu told the Post Office” such and such. Specific individuals are not identified, and they are certainly not called as witnesses to be asked questions about their approach, or (for example in respect of KELs) how they could have been so factually wrong in their explanation of the KELs’ contents. I have formed a dim view of the accuracy of information of this nature coming from Fujitsu, regardless of the person in that organisation responsible for communicating it. 646. Notwithstanding this, I wish to make one matter completely clear. If a witness refers to a particular document in their oral evidence, and the other party in this litigation wishes to see a copy of that document, then (absent issues of privilege) that ought to be disclosed, even more so if a specific request is made. This is by far the most sensible and proportionate way forward. It is in no party’s interests to engage in squabbling about relevance of a specific document in those circumstances. It costs far more to argue back and forth in solicitors’ letters than it does simply to give disclosure. In any event, the relevance of the document is made clear by the fact that it has been referred to orally in evidence. 647. It is also correct to say that notwithstanding the different burden upon the different sides in terms of disclosure in the Horizon Issues trial, which has undoubtedly been far heavier for the Post Office, this burden will not always be borne in these proportions. The Post Office also has the benefit of a far higher number of legal advisers. As at the date of this draft judgment, simply in terms of the total number of leading counsel engaged in the group litigation, the Post Office outnumbers the claimants by 4:1. In terms of the number of solicitors’ firms, the Post Office currently outnumbers the claimants by 2:1; although only one firm was acting for the Post Office on the Horizon Issues, and the second firm acted for the Post Office on its unsuccessful attempt to appeal the Common Issues. Summary on disclosure 648. The Post Office is, in my judgment, somewhat at risk of downplaying certain unsatisfactory aspects of its own disclosure that have emerged during the Horizon Issues trial. For example, the incident concerning the incorrect statements made to the court about the Royal Mail refusing disclosure without an order of the court was described in the Post Office’s closing submissions as a “mix up” which was “regrettable but should not be the subject of criticism of Post Office.” I do not consider telling the court something in express terms which is factually wrong can properly or correctly be identified as merely a “mix up”. Two points that were expressly explained by its leading counsel were simply not factually accurate. The Royal Mail was not refusing disclosure, and it did not require an order of the court before it would provide certain documents. It had not even been asked for them. 649. Similarly, in its closing submissions the Post Office referred to the interlocutory discussions about the Known Error Log as a “debate about relevance”. That again downplays what had happened, including the positive submissions made, that the Log was specifically not relevant and even if an order were made in respect of documents showing bugs, errors and defects, the Log would not be disclosed (even if it were in the Post Office’s control, which it was stoutly maintained it was not). The submissions made by the Post Office on the adequacy of its own disclosure have been rather undermined, since the trial ended, by the production (emanating from Fujitsu, once again) of over 5,000 KELs, something rightly described by the Post Office’s solicitors as “a serious matter”. 650. The claimants have not been entirely co-operative on their own disclosure, however. I consider that the criticisms of the claimants for serving what was described as “claimant specific” evidence are unjustified. At least one of the experts, Mr Coyne, adopted a specific approach and investigated specific issues or occurrences, and there is no reason that both of the experts could not have done so. Dr Worden did not, but that does not mean that the claimants were wrong to do so. Some of the criticisms made by the Post Office regarding the claimants are unjustified, therefore, but not all of them are. Although I do harbour the suspicion that the Post Office may have decided that, on disclosure at least, attack may be the best form of defence, some of the criticisms made in the Post Office’s closing submissions are justified. For example, for the claimants’ solicitors simply to state that “at present, it is not the Claimants’ intention to make another round of disclosure in relation to the Horizon Issues trial” as was done in July 2018, is not co-operative. Further, the reason that documents which Mr Tank had in his box, that were plainly relevant, were not disclosed was because Mr Tank had not been asked for them. Had he been asked – which is something that the claimants’ solicitors should have done, once it was known he would be a witness – these could and would have been disclosed. They certainly should have been. 651. The following points are of general application to both parties in this litigation. At some point a party who seeks to complain to the court about disclosure ought either to shelve its general disquiet on this subject, or make an application. 652. It is important to identify, so that the parties in this group litigation are aware, the court’s approach to disclosure. This is given for the purposes of guidance, and cost-effective and efficient progress of the group litigation going forwards. 1. Care must be taken when explanations or submissions are made to the court. If further time is needed in order for accuracy to be achieved, that time should be sought. Those giving instructions to counsel must be astute to ensure that they are giving correct information. 2. All the substantive issues will be resolved on the evidence, the applicable law, and taking account of submissions. A party does not obtain a preliminary advantage or “head start” on the substantive issues because of any criticisms about disclosure. 3. Criticisms about the other party’s disclosure should be proportionate. Depending upon the circumstances and the nature of the document, the fact that disclosure is given late might be relevant, but simply because continuing disclosure is given by one party does not mean that there is anything sinister lurking in the background. 4. The court can always be asked to resolve disputes about documents. Mr Coyne sought a large number of different types of documents, but no applications were made by the claimants even though these were not provided. Criticisms of the Post Office’s disclosure by the claimants has to be seen in the light that the claimants did not consider it sufficiently important at the time to make any applications to the court. This is not to encourage applications. Sensible and well-advised parties who co-operate will usually deal with almost all disclosure issues consensually. 5. The parties are expected to comply with their disclosure obligations. If a document comes to light that should have been disclosed but has not been, it should be provided promptly. If this is discovered actually during a trial, it must be disclosed very rapidly indeed. As happened with the document referred to at [618] above, a document concerning the Drop and Go bug was discovered actually during the Horizon Issues trial, then (as Mr Parsons put it in his explanatory witness statement) “for the next couple of months” kept by the Post Office’s solicitors while “research” was done. It was then disclosed just before resumption of the trial “in a batch”. This is not working to the time scale either required, or expected, by the court, and it is not the correct way to deal with disclosure during a trial. This is a laissez-faire approach to disclosure of an important document during the trial itself, and the document should have been disclosed very promptly. 6. Both parties will be held to the same standards. The Post Office has, so far, and given the issues that have been tried, borne the bulk of the disclosure burden. The claimants are expected to observe their disclosure obligations in the same way, and will be expected to do so in the future when further issues and the claimants’ different cases are tried in 2020. 7. Witness statements will be ordered from a party when the court requires an explanation. Nothing else should be read into the ordering of a witness statement, other than it means precisely that -- an explanation is required. 8. It is obvious that the Post Office has had to rely upon Fujitsu to a large degree. However, given it was Fujitsu who told the Post Office what the Known Error Log contained – see further [586] above – Fujitsu has, so far, shown itself not to be entirely reliable in this respect. Fujitsu are also responsible for the Post Office making a directly incorrect important statement in its EDQ about retention of KELs, which led to the disclosure of about 5,000 of these some months after the trial closed. 9. Adoption of the Model C procedure does not mean that parties can keep adverse documents to themselves and not disclose them until a specific request is made. Such adverse documents should be disclosed promptly. 10. Finally, disclosure is very expensive. The court will be astute to guard against it becoming either satellite litigation or a weapon in the interlocutory arsenal. 653. I have exhorted the parties to co-operation many times before in this group litigation. I will continue to do so. Eventually, it is to be hoped that they will emerge into the broad sunlit uplands of the over-riding objective on disclosure matters. G. The Experts’ Agreements 654. There were four experts’ agreements in this case. Experts’ agreements are very important. Permission to rely upon IT expert evidence was granted in the First CMC Order dated25 October 2017 . The two experts met on a number of occasions and agreed certain points. 655. As will be seen from Part H of this judgment, the experts approached their respective exercises in very different ways. However, they did reach some areas of agreement that were very helpful. Their joint statements or experts’ agreements did however also set out a considerable number of areas of disagreement. As the trial approached, I requested through the parties that the joint statements of the experts focus more on areas of actual agreement, rather than recitations of position and disagreements. 656. The 1 st Joint Statement was dated4 September 2018 and set out the following, inter alia, at the very beginning of the statement: “Each expert's approach to writing his report, and to this joint memorandum which foreshadows their reports, could broadly be one of three possible approaches: a) To focus mainly on negative points found in the disclosed documents about where Horizon may have fallen short. b) To focus mainly on those aspects of Horizon which were intended to achieve robustness and reliability, and the evidence implying that they succeeded. c) To provide the court with a clear foundation for understanding the design and operation of Horizon; then, building on that foundation, to provide a balanced assessment of the ways in which Horizon succeeded, whilst addressing any disclosed issues where Horizon may have fallen short.” 657. This was the first area of disagreement. Mr Coyne stated in the section immediately following, entitled Areas of Disagreement: “In my opinion the technical issues can be answered without reference to the Claimants high level allegations document. The issues are about how Horizon and its interactive components operated and the processes employed by Post Office and Fujitsu in supporting these systems and the data within. Whilst my report will take a balanced approach, it is the case that many of the issues require a deep focus on the occurrences of bugs, errors and defects as well as the potential for modification of transactional data. Whilst context will be provided as to how Horizon should work in typical circumstances, it is the non-typical operation where focus will be placed.” 658. This approach was not agreed by Dr Worden. He stated: “I intend to take the balanced approach (c). To provide a joint statement at this stage which will be useful to the court, under each Horizon issue the experts need first to agree what is worth agreeing having regard to the case which is being alleged. For instance, it would not help the court for the experts to list a number of detailed points as agreed, if one or the other expert thinks that those points are irrelevant or unimportant to the case alleged; or if collectively the points imply a view which, in the opinion of either expert, is not a balanced view. Therefore, the areas of agreement may at this stage be limited and more high level. My approach will also focus on the Horizon system. It will consider Horizon within its wider business context, but I do not believe that the Horizon issues extend to offering any opinion on the wider business practices of Post Office.” 659. I consider that the balanced approach, which both experts stated they would be adopting, was the correct one. However, I do not find the following statement by Dr Worden in this 1 st Joint Statement as helpful: “under each Horizon issue the experts need first to agree what is worth agreeing having regard to the case which is being alleged. For instance, it would not help the court for the experts to list a number of detailed points as agreed, if one or the other expert thinks that those points are irrelevant or unimportant to the case alleged.” 660. It is a matter for the court to consider how helpful a particular point of detail, agreed by the experts, is or is not to the issues to be decided. Obviously the parties can make submissions about that, but it is not for an expert to decide that a point of detail, upon which he may be agreed with his opposite number, should however not be agreed because that point will not be helpful to the court; or that it should not be included in a joint statement. It is also not for experts to direct the court in any particular direction. The Horizon Issues were carefully drafted by the parties, approved by the court, and specifically included in an Order. They were the issues that the experts were to agree, and detailed points of an IT nature that arose within those Horizon Issues should, where possible, have been agreed. 661. Because Mr Coyne and Dr Worden approached their tasks so differently, the former inspecting a number of PEAKs and KELs about 10 times greater in number than the latter, this meant that a great amount of very detailed matters shown within those documents were not agreed by Dr Worden. He approached his expert exercise from what he described as “a top down” approach, a methodology that found its most stark manifestation in his statistical analysis of likelihood in his Section 8. 662. I will include within this judgment, not the full text of all four of the different Joint Statements, but some lengthy passages. This is to identify the large degree of agreement between the experts. I will however omit (with some isolated exceptions) the passages where the experts set out their separate, not-agreed, views. The 1 st Joint Statement 663. I also quote at this stage from the 1 st Joint Statement, signed on4 September 2018 , and the points of agreement and disagreement in respect of Horizon Issue 1. This is because in my judgment it is an important point of agreement, in respect of an important Horizon Issue, but also it is useful to set out the approach within the Joint Statements. “2. Issues and Position 2.1 Issue 1 Issue 1 – To what extent was it possible or likely for bugs, errors or defects of the nature alleged at §§23 and 24 of the GPOC and referred to in §§ 49 to 56 of the Generic Defence to have the potential to (a) cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, or (b) undermine the reliability of Horizon accurately to process and to record transactions as alleged at §24.1 GPOC? Areas of Agreement: Evidence exists that bugs/errors/defects have caused actual discrepancies or shortfalls relating to Subpostmasters’ branch accounts/transactions. Each time any IT system (including Horizon) is changed there is the potential to introduce new bugs/errors/defects. Once bugs/errors/defects are discovered, they take time to resolve and therefore systems such as Horizon often continue to operate with bugs/errors/defects with or without workarounds in place. Theoretically, bugs/errors/defects that existed within Horizon have the potential to cause apparent or alleged discrepancies or shortfalls in relating to Subpostmasters’ branch accounts/transactions. In the event that any specific bug/error/defect had such an effect, the experts have differing views as to the ‘extent’ of the impact that such bugs/errors/defects may have had on branch accounts.” (Emphasis added) The statement continues: “Areas of Disagreement: Jason Coyne: As identified by select Known Error Logs (KELs), many thousands of bugs/errors and defects occurred and many of these known errors had impacts that recurred in different circumstances, therefore the reliability of Horizon to accurately process and record transactions is questionable. Such facts undermine the reliability of Horizon to accurately process and record transactions. Examples: (i) the Calendar Square/Falkirk bug – which led to Horizon failing to recognise transfers between different stock units, thereby affecting Branch accounts; (ii) the Payments Mismatch defect – which affected at least 62 branches and was related to the process of moving discrepancies into the local suspense account, thereby affecting Branch accounts. This defect was not capable of being identified by Subpostmasters, who would have believed from Horizon that their account was balanced when it was not; and (iii) the Suspense Account bug – which erroneously replicated suspense account items for 14 branches from 2010 in the same monthly trading periods in 2011 and 2012, thereby affecting Branch accounts. In my opinion there is possibility of bugs/errors/defects existing in Horizon that have not yet been discovered. Further, most bugs/errors/defects that exist in Horizon have existed for extended periods of time before they were discovered. I understand that Horizon had at least 19,842 ‘releases’ of new software, each one of these could have introduced new bugs/errors/defects. Dr Worden: My current preliminary view is that undetected bugs/errors/defects in Horizon were very unlikely to be the cause of permanent shortfalls or discrepancies in a branch's accounts. It is necessary to define measures for the 'extent' of issue 1. At least two measures are possible: the number of such bugs and errors, and their net expected financial impact on claimants' branch accounts, compared to the total shortfall experienced by all claimants, which is of the order of£18 million . I intend to assess both measures. The latter measure may be more useful, because if any set of bugs has expected financial impact much less than£18M , it can do little to account for the claimants' shortfalls, either collectively or individually. I propose to assess these measures from a number of different sources of information: (a) the defects of Horizon cited by the claimants in para 1.3 of the outline; (b) the KELs identified by the claimants in para 1.4 of the outline; (c) other KELs; (d) data on their shortfalls provided by the claimants in their schedules of information; and (e) data from other sources (if available). I am still scoping the data needed for (e). If it is available, I will provide it to Mr Coyne as soon as I have it. My preliminary analyses of the data on (a) - (d) imply that: • For the majority of KELs, (which record known bugs), it can be shown, from an understanding of the robustness measures built into Horizon or otherwise, that they would have no permanent impact on branch accounts. • The net financial impact on claimants' branch accounts of any other bugs, including undiscovered bugs, is very small compared to the claimants' total shortfalls. I am not yet able to quantify this, but I intend to do so.” 664. In my judgment, this presages Dr Worden’s statistical analysis of likelihood, which I deal with as part of Section I. The following text used by Dr Worden should be considered: “….if any set of bugs has expected financial impact much less than£18M , it can do little to account for the claimants' shortfalls, either collectively or individually” The phrase “much less than£18M ” could, potentially, direct expert consideration of Horizon Issue 1 away from what it is in fact asking, towards a different issue entirely. The experts were not tasked only with investigating bugs, errors and defects that had a proven financial impact of (say)£18M ,£10M , or any financial figure at all, and certainly not one in the millions of pounds. This was for the following reasons. It was never part of the Post Office’s position at the case management stage when the Horizon Issues were discussed and considered, that there should be any sort of threshold for financial impact, above which the experts should investigate, and below which they should not. The wording of Horizon Issue 1 has already been addressed above, and this uses the terms “possible or likely”. This notion of imposing a financial threshold must have come from Dr Worden. 665. Nor is the litigation intended to be a wide-ranging enquiry into all of the Post Office’s accounts with all of its SPMs over the 15 or 16 year period in question. The Post Office’s position is that Horizon is sufficiently robust such that it cannot be the cause of the different shortfalls complained of by the hundreds of different claimants. The counter-position to that by the claimants is not that particular discrete sets of bugs had, individually, an impact of millions of pounds upon their branch accounts. If that were the claimants’ case, then Dr Worden’s approach as set out in this joint statement might be understandable. Rather, the claimants’ case is that there are a large number of different bugs, errors and defects that each affected each of their branch accounts over time, with the end result being that each of them was (in accounting terms) identified as owing the Post Office sums in the tens of thousands, or thousands of pounds (or less) with the consequences which are well known. Some discrepancies were only a few pounds, but are said to have occurred many times. Some were said to be tens of thousands of pounds cumulatively. Some occurred many times; others less often, but with greater individual effect. One bug – of which more later – is said to “double up”, which would mean an exponential effect as a loss doubles, then quadruples and so on. If one imposes, at the beginning of the expert investigation process, a filter or threshold that looks for a “set of bugs” causing losses of “millions of pounds” that will have the effect of restricting the field of investigation. There is no reason for such a restriction, in my judgment. 666. On Horizon Issue 2, the experts were agreed, so far as that issue was concerned, that the extent to which any IT system can automatically alert its users to bugs within the system itself is necessarily limited, and while Horizon has automated checks which would detect certain bugs, there are types of bugs which would not be detected by such checks. The experts were also agreed, in respect of Horizon Issue 6, that whilst Horizon contains measures and controls for detecting system integrity concerns, the automatic mechanisms have failed in the past. The experts did not agree as to the ‘extent’ of prevention, detection, identification, reporting or risk reduction that the automatic and manual control measures delivered; however, the agreement that automatic mechanisms have failed in the past, which was contained in the 1 st Joint Statement, is in my judgment important. It means that the Post Office cannot rely upon the presence of automatic mechanisms or processes as being an answer to the consequences of failures in system integrity. 667. It was also agreed by the experts that the causes of some types of apparent or alleged discrepancies and shortfalls may be identified from reports or transaction data available to the SPMs, but that other causes of apparent or alleged discrepancies and shortfalls may be more difficult or impossible to identify from reports or transaction data available to the SPMs in question, because of their limited knowledge of the complex back-end systems. Identification required the cooperation of PO staff and the SPMs. Again, I consider this expert agreement to be important. This was, essentially, the essence of the complaints raised by Mr Bates himself in the very earliest days of Legacy Horizon, namely that there was or were insufficient information or reports available to him to enable him to identify the causes of apparent or alleged discrepancies. The experts are agreed that, for some causes of apparent or alleged discrepancies this is indeed the case, due to the complexity of the back-end systems (back-end means not at the branch) and the limited knowledge a SPM would have. 668. The experts agreed that Horizon had evolved, and its robustness may have varied throughout its lifetime. They also agreed that the level of robustness may have increased or decreased as the system was changed. The existence of branch shortfalls was agreed, but they were not agreed at the stage of the first joint statement that this indicated any lack of robustness. They were also agreed, as at the date of their 3 rd Joint Statement, that Horizon as it is in 2019 is relatively robust. However, given the span of the years under consideration is 2000 to 2010 for Legacy Horizon, and 2010 to date for Horizon Online, and given the agreement to which I have just referred that robustness may have increased or decreased, this only takes one so far. 669. On Horizon Issue 4, the experts agreed that there are a number of actual reported errors in data recorded within Horizon arising from (a) data entry, (b) transfer or (c) processing of data. Therefore, the potential exists. The experts do not agree as to “its “extent”.” Dr Worden saw this issue as a subset of Horizon Issue 3. He felt that because of the measures “built into Horizon”, errors of data entry, transfer and processing were very unlikely to affect branch accounts. The experts agreed on Horizon Issue 6 that the automatic measures and controls for detecting system integrity within Horizon had failed in the past; they could not agree as to the extent that the automatic and manual control measures delivered. Dr Worden also made the point that bugs, when discovered, could not be fixed instantly and what he called “trade offs” were required when dealing with practical release management. 670. So far as remote access and Horizon Issue 7 was concerned, at the time of the 1 st Joint Statement Dr Worden was unclear about this and the technical document provided to him by Mr Coyne had not been received in sufficiently timely manner such that he could consider it. By the time the Horizon Issues trial had started, the Fujitsu witnesses such as Mr Parker had corrected their earlier incorrect witness statements and accepted that this could be done. However, those statements were not served until November 2018 so would not have been in existence as at the time of the 1 st Joint Statement. 671. On Horizon Issue 9, the experts agreed that the causes of some types of apparent or alleged discrepancies and shortfalls may be identified from reports or transaction data available to SPMs. Other causes of apparent or alleged discrepancies and shortfalls may be more difficult or impossible to identify from reports or transaction data available to SPMs, because of their limited knowledge of the complex back-end systems. Identification required the cooperation of PO staff and SPMs. 672. On Horizon Issue 10, it was agreed that by the very nature of rolling out fixes in any IT system, including those implemented by Fujitsu, these had the potential to affect transaction data or data in branch accounts. This could be seen as stating the obvious, but at least the experts agreed it. The experts were also agreed that the use of tools and facilities to do this should be auditable, however the maintenance of logs recording this would be dependent upon retention periods and the size of the logs in question. Again, this could be seen as stating the obvious. The experts were not agreed on Issue 12, how many times the ability to do this was used, if at all. Nor was there any agreement on Issue 13, the extent to which such facilities had the potential to affect the reliability of branch accounts; Issue 14, a detailed issue with 7 sub-issues concerning functionality; or Issue 15, which concerned Transaction Corrections or TCs. 673. The experts agreed, so far as the existence and use of permission controls is concerned, which is the essence of Horizon Issue 11, that the usage of tools and platforms which existed specifically for the purpose of accessing and modifying transaction data, should be auditable. It was agreed however that the maintenance of logs would be dependent upon retention periods and size. Such tools which were identified initially in the 1 st Joint Statement under Horizon Issue 10 areas of disagreement by Mr Coyne, were at that stage split into the following: 1. Global Branches which would enable the input of transactions within Horizon as though it had come from an actual Branch; 2. the Branch Transaction Correctional tool; 3. the Transaction Information Processing repair tool. 674. The best description of “Global Branch” or “Global User” is contained in a Post Office document headed “Global User Process Management” which said that it allowed people such as auditors “to be able to go out to any branch and use the same log on to access any Horizon kit in Branch without having to request a single use password at each branch. The individual requiring a Global Use account has to be approved by a Manager showing on the Global User requester and approver list”. However, the document also identified that PWC had discovered that “the current Global User list inherited from Fujitsu contained employees who had left the Business. This raised concerns over the security of access to Horizon. Although this is a low risk in terms of Horizon access (the individual would have to gain access into a Branch first) it is still a risk and needed to be mitigated.” There were 32 such former employees. It was in respect of the permission control procedures concerning these tools that the documented authorisation process was identified, using documents called the “Master Service Change” or MSC documents and “Operational Control Procedures” or OCPs. These documents were requested by Mr Coyne but no actual disclosure application was issued. The document to which I have referred explaining Global User was in the trial bundle. The 2 nd Joint Statement 675. The 2 nd Joint Statement is dated25 February 2019 and identified and collated the different bugs, most of which were given actual names, which had either been identified and agreed, or which Mr Coyne maintained were bugs. This statement was in landscape form and lengthy, and included what became to be referred to as “the bug table”. I adopt the terms used by the experts for the descriptions of the different bugs, or alleged bugs. The number of different bugs that were agreed by Dr Worden, by the time the trial ended, was 11. The description of this in the joint statement itself was “The structure of the document captures i) A table of bugs/errors/defects containing evidence of financial impact upon branch accounts that both experts agree (or indicate if they do not), ii) all expert agreements grouped by Horizon issue, and iii) additional comments and observations input by the respective expert.” 676. Prior to the commencement of this litigation, the Post Office had acknowledged the existence of only two bugs in Horizon. Mr Coyne’s evidence was that there were 29 relevant to Horizon Issues 1 and 4, and he clarified in his cross-examination that the number that he considered had lasting impact upon branch accounts was 21. I deal with that cross-examination at [794] to [798] below. 677. The 2 nd Joint Statement also said in its Introduction that: “Because of time pressures and the complexity of the issues, we have not been able to address all the Horizon issues in this joint statement. We will issue a subsequent joint statement addressing those issues we have not yet addressed. For those issues we have addressed in this statement (issues 1, 2, 9, 14 and 15), the layout and references are not as polished as we would have wished.” 678. It is extremely useful for the court to have Joint Statements from experts, and I am grateful to the experts for producing these statements, in particular what was called “the bug table” in the 2 nd Joint Statement, in what is a very complex case, and when they were under considerable time pressure. The bug table was very widely used in both cross-examination by the parties and indeed in their submissions. The so-called “Bug Table” 679. The bug table, and my findings upon the bugs within it, are in my judgment central to the Horizon Issues. Evidence concerning bugs in the Bug Table formed a large part of the expert evidence that was tested in cross-examination. The different bugs, or alleged bugs, identified in the Bug Table were as follows. I have added to the following the summary of the position of the Post Office in Appendix 2 of its Closing Submissions, which in some cases accepted there was such a bug, and qualified that acceptance in most instances by references to what was said to be the impact of the bug. I deal with detailed findings on those bugs that are in issue in the Technical Appendix to this judgment. For the detailed submissions by the Post Office for each bug, the paragraph numbering in Appendix 2 started again at 1 for each bug. This did not cause particular difficulty, and because the Post Office explained that Appendix 2 had been put together by different parts having been drafted by different counsel/solicitor teams, this is understandable. I refer to it here, because otherwise a reader of the passages below might become confused as to why there seem to be so many paragraphs all (for example) numbered 2 in Appendix 2. The following summary of the Bug Table is intended to be a useful summary for this judgment, and is not restricted solely to the entries within the 2 nd Joint Statement. 1. Receipts and Payments Mis-match bug. This was agreed in the bug table as an “acknowledged bug” which had an impact on branch accounts. Dr Worden added in his comments that “Therefore, the extent of this bug is well established, in the GJ analysis.” GJ is Gareth Jenkins. The identified year of its effect was 2010. It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. In the summary in Appendix 2, the Post Office stated that “In the event, however, this bug resulted in transient impact only.” 2. Callendar Square/Falkirk bug. This too was agreed in the bug table as an “acknowledged bug” which had an impact on branch accounts. Its identified years of effect were 2000 – 2006. It is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had “transient impact”. Dr Worden added different comments, including that “the bug arose from a fault in the underlying Riposte software, so it is not surprising that it took Fujitsu some time to understand it, or that they had to rely on the suppliers to fix it. It does not show poor system design or support by Fujitsu”. That latter sentence is a little surprising, and seemingly very defensive on Fujitsu’s behalf. This is because the Horizon Issues are all about the operability and functionality of Horizon, not who is to blame for the presence of any bugs, errors or defects. Riposte software was part of Legacy Horizon. The fact that it is a product designed and supplied by Escher, and not Fujitsu, is not relevant to the dispute between the Post Office and the claimants. In any event, and as accepted and recited in paragraph 2 of the Summary of Appendix 2 dealing with this bug, “Mr Coyne asserts that Bug 2: Callendar Square is a bug with lasting financial impact and in JS2, Dr Worden appears to agree that there is strong evidence of this….” 3. Suspense Account bug. This was agreed in the bug table as an “acknowledged bug” and its identified years are 2010 to 2013. It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. Dr Worden’s comments do not include that it had an agreed impact upon branch accounts. This is because of a concept he introduced (expanded in his reports) of “transient effect”. What this meant was that if there was an impact on branch accounts by something within Horizon, but that was then corrected by a TC, he concluded it had a “transient effect” upon branch accounts and dealt with it differently. I address this concept below. Other comments of his in relation to this acknowledged bug are: “It was a transient effect arising not from a fault in the software, but from a change in database archiving policy in 2010. The delay in correcting it arose from a failure of communication between PO and Fujitsu. Because the bug would only manifest itself annually for any affected branch, the effects of this delay were not widespread. Peak PC0223870 shows that Fujitsu were able to identify the branches affected, even when Subpostmasters did not report it. There is evidence that the branches were compensated, as I would expect from the normal error correction processes.” 4. Dalmellington bug/Branch Outreach Issue. This was not acknowledged as a bug in the bug table, but is effectively accepted by the Post Office as a bug. It is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had “transient impact”. Indeed, for some time during the trial the Post Office would refer to this as “an issue”. I find that it is a bug – that much does not seem to be controversial, as paragraph 2 of the detailed part of Appendix 2 relating to this bug states the following: “This is a bug which Mr Coyne states has lasting impact on branch accounts. Post Office submits that there was no lasting impact on branch accounts.” The years of its effect were 2010 to 2015. It is named in respect of the branch where it was “discovered” in 2015, although the investigation undertaken by Fujitsu in that year identified previous occurrences. There were over 110 occurrences of it over this 5 year period (the actual number is probably 116, but at least 114) with 88 branches affected. Some were impacted multiple times. It is what is called a “cash remming error” and only happened in certain circumstances. It was explained further by Mr Godeseth in his evidence at [445] onwards above. It would, in my judgment, have an impact upon branch accounts – it is the extent of that impact that is in issue, whether it was lasting or transient. Dr Worden relied upon what he described as “a well-tested process of reconciliation and TCs to detect and correct errors in cash remming (used 20,000 times per year)” and said it was “straightforward for Horizon to detect any discrepancy between a “rem out” and the corresponding “rem in” (a mismatch arising either from a miscount, or a multiple count of a pouch) and then a TC can be issued.” He also added that “this process catches and corrects remming errors, whatever their cause - including if they arise from, or are provoked by, software faults.” He therefore implicitly accepted that the Dalmellington bug was a software fault, although he did not say so in terms. I find that it was plainly was. He was also reliant upon the process of TCs to correct it. Further, the bug was there for 5 years and was not discovered, although its effects doubtless were. I find that this was a software bug that impacted upon branch accounts and I will come to the nature of its effect in my findings on the bugs that are not agreed. 5. Remming In bug. This was not acknowledged as a bug in the bug table. However, Dr Worden accepted it was a bug or defect in his cross examination and it is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is again said to have had transient impact. The years it was said to have been present in terms of effect were “March – August 2010 and recorded as fixed approx. 2011”. Mr Coyne said there were 14 branches affected. Dr Worden’s entry in the Joint Statement relied upon TCs, and stated “As for the Dalmellington bug, above – PO had a robust process for detecting and correcting remming errors, whatever their origin. So, there were no lasting effects on branch accounts .” In paragraph 2 of the detailed part of Appendix 2 relating to this bug, the Post Office submitted that “ any discrepancy would be transient as instances of this bug are caught by automatic reporting.” (emphasis added) The answer to this is therefore dependent upon my answer to the point concerning lasting and transient impact. 6. Remming Out bug. This was split into two in the bug table, 6(i) which was identified in KEL acha508S and 6(ii) which was identified in KEL GMaxwell3853P. The former was identified in February/April 2007, recorded as fixed approx. 2007, with Mr Coyne identified 57 branches affected; and the latter was in May 2005 with one branch being affected. They were both remming out issues, hence the grouping by the experts in the bug table. It was not acknowledged as a bug in the bug table. However, it is now accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. Again, for both of these iterations, 6(i) and 6(ii), Dr Worden used the same wording as he had in the entry for bug 5, the Remming In bug, namely “As for the Dalmellington bug, above – PO had a robust process for detecting and correcting remming errors, whatever their origin. So, there were no lasting effects on branch accounts” (emphasis added). In paragraph 2 of the detailed part of Appendix 2 dealing with this bug, the Post Office stated that this “ comprises two separate issues, only one of which was a bug. Any discrepancy caused by either issue would be transient as instances of both issues were caught by automatic reporting.” It was also said that Mr Coyne had conflated two unrelated issues under the heading “Remming Out”. Given this was an entry in the 2 nd Joint Statement agreed by the experts, it is a little unfair to state that Mr Coyne had done this, as the entry was obviously agreed by Dr Worden. However, it was split into 6(i) and 6(ii) and I will consider each separately. 7. Local Suspense Account issue, not the same as 3. Suspense Account bug. This was reported in 2010 and recorded as fixed in September 2010. It was not acknowledged in the bug table, but is now accepted as a bug by the Post Office in paragraph 6 of Appendix 2, although it is one of those bugs said to have had transient impact. Mr Parker had identified 33 branches affected. Mr Coyne recorded what he said were four associated KELs, namely acha5259Q (for which there were 6 PEAKs)(this KEL is mistakenly recorded twice in column 3, with acha5838T only mentioned in the text in that column); cardc2043L (10 PEAKs); PorterS199P (3 PEAKs); and acha5838T (which states there are “two different but similar problems” and appears in the text, but not in the list of KELs at the end of the text by Mr Coyne). Dr Worden’s comments were that “The KEL acha5259Q implies that PO and Fujitsu were able to identify all occurrences of the problem, without being notified by any Subpostmaster. I would therefore expect them to have corrected any impact on branch accounts as part of normal error correction processes. I would not expect evidence of all corrections to accounts to have survived to the present day. PEAKs and KELs are not used to record corrections of financial impact.” He also relied upon a statement by Mr Parker that there was ‘Temporary financial impact which would have been cancelled out in the following period by a corresponding discrepancy’. This introduces another concept, similar if not identical to Dr Worden’s “transient impact”, and that is “temporary financial impact”. Dr Worden is correct that PEAKs and KELs do not record corrections of financial impact; they do however record financial impact, because when an SPM reaches SSC (which raises the PEAK) they often start by recording “SPM says he/she has a problem in that……” and financial impact is often then recorded. The Post Office in paragraph 2 of the detailed part of Appendix 2 dealing with this bug states that “any discrepancy would not be lasting”. This does therefore accept, albeit implicitly, that it would have an impact upon branch accounts. It is also submitted that it was a “teething problem” from the early days of the Horizon Online pilot scheme. 8. Recovery Issues. The Post Office does not accept these are bugs at all. These are not agreed as bugs by the experts, and there are four different types included in the table, with years of effect from 2010 to 2018. Mr Coyne’s entry stated that “The text within the PEAKs and KELs suggests that in each case a branch account discrepancy would be evident and would require correction by the Post Office.” Dr Worden stated that: “The KELs and PEAKs cited by Mr Coyne are not indicative of errors in Horizon. They provide guidance on how to correct discrepancies caused by human errors or other errors in transaction recovery ('recoverable transactions') Because there were many such errors, there were many calls to the help desk and many PEAKs and KELs. Normally, correction of errors involved back office reconciliation and issuing TCs. This was accurate and effective; I have derived an upper limit of£2 per branch per month on the mean impact of erroneous TCs. One important KEL acha959T was guidance to the back office MSU, not for Subpostmasters” There was a minor typographic error in the submissions in paragraph 4 of the detailed part of Appendix 2 which referred to bug 9. However, the four different types of recovery issues are addressed in the subsequent paragraphs and the conclusion paragraphs deal with bug 8. This was later corrected in a sheet of corrections, which clarified that paragraph 4 of the detailed part of Appendix 2 should have stated "
“ There was a bug which has potential impact on branch accounts , early in the lifetime of Horizon. Soon after it arose, the error was trapped and detected by DEP and was then soon fixed. The fault was easily noticeable at branches before the error trapping which was soon introduced and would be even more noticeable after that. Only three branches appear to have been affected, as described by Mr Coyne. Because it was so noticeable at the branch, and the Peak is concerned with a software error rather than any other cause , I would expect any discrepancies in branch accounts to have been corrected.”
“Bug 14: Bureau Discrepancies is a bug with the potential for lasting financial impact. There are two distinct issues which fall under this heading. With regards to the first issue the branch was made good and a fix was implemented. The second issue was not a bug in Horizon nor an issue which could have impacted branch accounts; it created what was essentially a cash flow problem for the branch.”
“Post Office accepts that Bug 18: Concurrent Logins had a potentially lasting financial impact. There is no evidence of any discrepancy in the PEAKs referred to by the experts.”
“discrepancies could occur - manifesting themselves as a receipts/payments mismatch. This had the potential to affect branch accounts. The mismatch would bring it to the attention of the Subpostmaster, who would require it to be investigated, except possibly in the case of small mismatches, which he might pass off as an error in the branch (e.g. of counting stock). “….Fujitsu believed it was a problem with the underlying Riposte software, and passed it to Escher. In September 2000, the problem was 'Now formally fixed in Build 223 update 19 which was released overnight.' However, the new release from Escher did not, as it was expected to, fix the problem. Escher denied that is was a bug in Riposte, but Fujitsu believed in July 2001 that 'This is clearly a bug in the Supplier's code'.”
‘Impact small until bug fixed - rounding errors 10 -5 in exchange rates.’
“To what extent was it possible or likely for bugs, errors or defects of the nature alleged… to have the potential to (a) cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, or (b) undermine the reliability of Horizon accurately to process and to record transactions.” 683. The use of the words “possible” and “potential” in this issue makes it clear, in my judgment, that the number of “bugs, errors or defects” that fall to be considered is, even on the Post Office’s own submissions, 21 different bugs. I find that to be a significant total. 684. Part only of Appendix 2 to the Post Office’s Closing Submissions dealing with bug 16, Reconciliation Issues, was, by administrative error, not submitted by the date for closing submissions, or until some time after oral submissions were delivered. The Post Office’s solicitors realised this in September 2019. There were three missing pages. I am satisfied that these were drafted prior to the date for service, and I allowed these to be submitted late, together with a clarification document, and short responsive written submissions by the claimants (as the claimants had been deprived of the opportunity to address these three missing pages orally in July). I consider that I am permitted to do this under the court’s general case management powers, and that to do this was to adopt a reasonable and proportionate course. The Post Office’s Closing Submissions was a very substantial document, and it was in my judgment sensible to admit the 3 pages only that had been omitted from one of the appendices. 685. Other elements of the 2 nd Joint Statement were also agreed. These included that Horizon had produced over 3 million sets of monthly branch accounts (a point relevant to Dr Worden’s Section 8 statistical exercise), and also that the experts had different views on “branch impact”
“Mr Coyne refers to any discrepancy that caused a loss (or gain) within branch accounts that needed corrective action as an “impact to branch accounts”
“2. Within Horizon Online, a Balancing Transaction can be created using the BRDB Transaction Correction Tool (called BRDBX015). 3. In the one acknowledged Balancing Transaction which has been disclosed, there were SQL INSERTS into only two tables of the BRDB. These tables were ops$brdb.brdbb_rx_eposs_transactions and ops$brdb.brdb_rx_rep_session_data” 4. Mr Coyne stated the following, as one of a number of passages that were not agreed. “Outside of the Transaction Correction Tool other methods were available in Horizon Online. The APPSUP role used by Fujitsu has elevated privilege and could be used “for un-envisaged ad-hoc live amendment not covered by BRDBX015” as outlined in the referenced document. Fujitsu also has the ability to perform balancing transactions via direct SQL operations (using a command line interface) to perform corrective transactions in Horizon Online (including deletions of operational data). Such modifications may or may not be visible to the Subpostmaster and they could be done without their consent.”
“Dealing with a case justly and at proportionate cost includes, so far as is practicable: (a) ensuring that the parties are on an equal footing; (b) saving expense; (c) dealing with the case in ways which are proportionate – (i) to the amount of money involved; (ii) to the importance of the case; (iii) to the complexity of the issues; and (iv) to the financial position of each party; (d) ensuring that it is dealt with expeditiously and fairly; (e) allotting to it an appropriate share of the court’s resources, while taking into account the need to allot resources to other cases; and (f) enforcing compliance with rules, practice directions and orders.” 736. Since the Civil Procedure Rules, trials are usually set down for a fixed time estimate and are expected to be time-tabled to finish within that period. The court system does not have the luxury (if luxury it is) of giving endless periods of time to litigants to argue their cases to the n th degree. This case, as with all litigation, is very important to those involved in it – to all of the claimants, but also to the Post Office, as an institution generally, and also to its other directors, employees, SPMs who are not claimants and its clients too. The issues and different causes of action in both claims and counterclaims are of considerable importance, and the amount of money at stake is not small. However, the amount of money is not large in terms of specialist litigation, for example. It is not measured in the billions, or even many hundreds of millions. Ignoring claims for general damages, the amount of damages that are specified in liquidated sums are said to be about£18 million (although that is only an approximate figure currently). There are, of course, also reputational issues at stake as well as money. 737. Judgment (No.5)[2019] EWHC 1373 (QB) made it clear at [78] that the total of the parties’ reported costs as at the date of that judgment was over£25 million (£12.6 million for the claimants, and£12.8 million for the Post Office). Since then, on25 June 2019 the Post Office notified the court that its costs had exceeded£13.9 million . 738. The reason for explaining this in detail is to put into context the following points about the scope of expert evidence, and the number of trial days available for the experts to be cross-examined. At the Pre-Trial Review, the Post Office submitted that at least four days was needed for the cross-examination of Mr Coyne. It was realistically accepted that longer than that was not likely to be permitted, although leading counsel for the Post Office did submit that he would prefer longer. The claimants submitted that they also wanted four days, but had trimmed that period to three days, for the cross-examination of Dr Worden. Given there were further experts’ meetings to come, and given my view of the scope of the expert material and considering the overriding objective, I set a trial timetable that permitted each side two and a half days to cross-examine the other side’s expert. That would have given a total number of five days of cross-examination for all the IT expert evidence. At that time, I considered that would be sufficient, and also proportionate, taking account of the overriding objective. 739. However, the subsequent Joint Statements following the PTR did not reach as much agreement as I had (perhaps naively) anticipated. I therefore adjusted the trial timetable and set a new one, with the Post Office being given four days to cross-examine Mr Coyne, and the claimants being given three days to cross-examine Dr Worden. That trial timetable also allowed an interval of one non-sitting week, to follow the evidence of fact, for each side properly to prepare their time-limited expert cross-examination in detail. 740. In my judgment, the period of time I had allotted at the PTR of 2½ days each was sufficient in itself; but the increased period of time allowed was undoubtedly sufficient for each party properly to cross-examine and test the expert evidence adduced by the other side in the litigation. Seven court days for the total cross-examination of one expert discipline in a case such as this is, given the Horizon Issues in this trial, generous. Counsel might not have had time to put each and every single individual point to the expert on the other side, but they had more than enough time to put the main ones (and a considerable number of subsidiary ones too, for that matter). Each side had leading counsel, well-versed in all of the issues, who have been involved in the case from at least the making of the GLO itself in early 2017, as well as a team of other counsel assisting too. Considerable pre-reading had been done by the court. As it turned out, the issuing of the recusal application on21 March 2019 led to an even longer period of the trial not sitting, with only a half day (for Mr Parker) of the trial taking place between21 March 2019 and4 June 2019 , a period of 75 days, or over ten weeks. The Post Office had a different counsel team acting on the recusal application than the team acting on the Horizon Issues trial. That 75 day period was a substantial period of time for each side to prepare detailed cross-examination on all, and any, points that they considered important, prior to the resumption of the trial and the commencement of the expert evidence on4 June 2019 . 741. The amount of the costs burden explained in [737] above has been incurred on the basis of the number of trial days made available. Obviously, further time for cross-examination would have led to even higher cost figures. There was a sufficient and proportionate amount of time at trial made available for the expert evidence to be properly challenged and tested, and there was more than sufficient time available for preparation of the time-limited cross-examination. I have set this out in detail in order to make it clear that the time limited nature of the Horizon trial, and the number of days allotted for cross-examination, has in my judgment had no effect on my findings on the Horizon Issues, or restricted either party’s opportunity to put its own case and test the case of the other party. 742. The other feature of time-limited cross-examination, given the depth of experience of counsel, is that I permitted considerable latitude to the cross-examiners. If counsel wanted to put the same point to an expert again and again; or spend an hour on one specific sentence or detail, when the same point could have been accomplished in five minutes; or devote significant time to high-level issues and avoid descending to specifics; then that was a judgment call made by that counsel. With more junior counsel, in another case, the court might discourage excessive time on less important or agreed points, to encourage progress; but here both sides had highly experienced leading counsel. I therefore intervened rarely. The amount of time made available in the trial timetable for each side to cross-examine the other’s expert was set down in advance, and if one side wished to devote (say) half a day to having an expert state, and then restate, an agreed position on a particular point, then that was very much a matter for them. Such an approach would lead to that period not being available to allocate to other subjects, but in my judgment this case in particular benefitted from allowing the parties to conduct their cross-examination, more or less, in the direction they wished. 743. I did, from time to time, seek either to have clarified what particular Horizon Issue was the subject of the cross-examination, or to reference it to particular other subjects. However, this did not occur very often, and the parties were permitted to use their allocated time as they wished. 744. Each side, both in their cross-examination and closing submissions, criticised the other side’s expert for not being sufficiently independent. This was done in different terms and in different ways, but the upshot of it was broadly the same. It was that their expert’s evidence should be preferred to that of the opposing side, for reasons of methodology and approach, as well as for reasons of detail. 745. The court, when faced with opposing experts, might dismiss the views of one expert entirely on the basis that they had not complied with their duties to the court. However, it is not necessary to do that, in order to prefer the views of one expert over another. There may, depending upon the subject matter, be room for opposing expert opinion, and it is simply a case that the court accepts one opinion for a variety of reasons. Also, in some cases, it might be that one expert’s view is accepted on some points, and the opposing expert’s views on others. 746. It is also not necessary to resolve every single point of detail between experts to arrive at the answers to the Horizon Issues. However, given these criticisms of the experts have been made, it is necessary to address them. This is because it is only fair both to the experts, and to the parties, to decide whether each expert has been sufficiently impartial in the exercise upon which they have been engaged such that the court can reliably base its answers to the Horizon Issues upon the conclusions reached by the experts. Impartiality and independence of experts is also of considerable, if not central, importance to the administration of justice generally. 747. The case that is most often stated in the context of the importance of independence on the part of experts is National Justice Compania Naviera SA v Prudential Assurance Co Ltd[1993] 2 Lloyd’s Rep 68 (Comm Ct), but it is almost always referred to by the name of the ship involved in the case, The Ikarian Reefer , which ran aground off Sierra Leone and then caught fire, becoming a total loss. Lord Woolf MR, the author of the Civil Procedure Rules, made clear in Stevens v Gullis[2000] 1 All ER 527 , 533 thatCPR Part 35.1 and its accompanying practice direction “did no more than reflect the position as it had been well enunciated in the authorities prior to the CPR coming into force”
“The principles that govern expert evidence must be carefully adhered to, both by the experts themselves, and the legal advisers who instruct them. If experts are unaware of these principles, they must have them explained to them by their instructing solicitors. This applies regardless of the amounts at stake in any particular case, and is a foundation stone of expert evidence. There is a lengthy practice direction toCPR Part 35 ,Practice Direction 35 . Every expert should read it. In order to emphasise this point to experts in future cases, the following points ought to be borne in mind. These do not dilute, or change, the approach in The Ikarian Reefer. They are examples of the application of those principles in practice. 1. Experts of like discipline should have access to the same material. No party should provide its own independent expert with material which is not made available to his or her opposite number. 2. Where there is an issue, or are issues, of fact which are relevant to the opinion of an independent expert on any particular matter upon which they will be giving their opinion, it is not the place of an independent expert to identify which version of the facts they prefer. That is a matter for the court. 3….. 4….. 5. Where late material emerges close to a trial, and if any expert considers that is going to lead to further analysis, consideration or testing, notice of this should be given to that expert's opposite number as soon as possible. Save in exceptional circumstances where it is unavoidable, no expert should produce a further report actually during a trial that takes the opposing party completely by surprise. 6. No expert should allow the necessary adherence to the principles in The Ikarian Reefer to be loosened. It is to be hoped that expert evidence such as that called by ICI in this case, and also in Bank of Ireland v Watts Group plc , does not become part of a worrying trend in this respect. There are some jurisdictions where partisan expert evidence is the norm. For the avoidance of any doubt, this jurisdiction is not one of them. Not only experts, but the legal advisers who instruct them, should take very careful note of the principles which govern expert evidence.” 749. The Ikarian Reefer , and the requirements ofCPR Part 35 , are the touchstones of expert evidence in this jurisdiction. It is of the utmost importance that they are adhered to both by experts themselves, and also by those who instruct such experts. Mr Coyne 750. Mr Coyne is a Partner of IT Group UK Ltd, and has 30 years of experience in IT. He has designed, programmed and supported Electronic Point of Sale (or EPOS) systems for use in payment handling, stock control and distribution in addition to full business accounting systems. He has also assisted public sector councils with revenue and benefits processing systems, investigated failures within stock market trading systems, European gambling and gaming networks and dealt with fraud in retail banking, point of sale, cash in transit and electronic funds transfer systems. 751. He is a member of the Academy of Experts, the British Computer Society and the Society of Computers and Law. He has an Expert Witness Diploma from Cardiff University Law School, achieving this in 2008. He has been instructed as an expert witness in both civil and criminal cases over the last 19 years, including by both UK and International firms of solicitors, in connection with technology project fault analysis, software development and digital forensic investigations. He has appeared before different tribunals both in the UK as well as overseas. He served two detailed reports with numerous appendices. He also signed four Joint Statements with Dr Worden. 752. Both the experts agreed in the 3 rd Joint Statement that Horizon as it is now is relatively robust. The passages in that Joint Statement are that: “in line with most other large-scale computer systems, Horizon's robustness has generally improved. From our experience of other computer systems, Horizon is relatively robust. We agree that 'robust' does not mean infallible and therefore Horizon has and will continue to suffer faults. Robustness limits the impact of those faults and other adverse events. This increase in robustness has, in part, developed from Post Office discovering bugs/errors and defects in live use and then applying fixes and improving monitoring.” 753. I consider that this is an important passage of the experts’ agreement, and indeed robustness is an important aspect of the Horizon Issues. It makes clear that, in part, it is the discovery by the Post Office of bugs, errors and defects, the application of fixes (which means changes to the software by means of a software fix) and the improvement of monitoring that has led to the increase in robustness that has been achieved in the system as it is at 2019. This inevitably means that in earlier times, Horizon was less robust, or not as robust as it is now. This applies both to Legacy Horizon and also to Horizon Online. There are undoubtedly more modern or recent bugs in Horizon, as can be seen from the Technical Appendix. Drop and Go, for example, was not even a product until about 2013, and the relevant bug in the Bug Table which is bug 28 occurred in July 2017. That caused an impact to the branch account in question of£100 ; modest or tiny in comparison to the total Drop and Go business for that year, or even that month, but still£100 which appeared as a deficit in the branch accounts, and still£100 which the SPM had to deal with. Until it was discovered the bug could not have been fixed. It would not have been present prior to 2013 (because Drop and Go did not exist) and any fix cannot have occurred prior to July 2017, if not a month or two later. The agreement that Horizon is relatively robust as at 2018 and 2019 is something that is doubtless of great importance to the Post Office, as it affects how the Horizon system as it is at the date of this judgment is both used by the Post Office, and perceived by the SPMs who use it every day. The Horizon Issues however deal with Horizon from the year 2000 onwards. 754. The other aspect of this part of the experts’ agreements that bears consideration is that the Post Office have to know about bugs, defects and errors in order that they can agree with Fujitsu that fixes have to be written and applied. This makes it particularly important, in my judgment, that Fujitsu were open and transparent with the Post Office about the presence of software bugs, errors and defects. Given Horizon was being used by the Post Office as the accounting procedure between it and its SPMs, there is every reason that Fujitsu would (or should) have to make the Post Office aware of bugs, errors and defects. 755. A point that was put to Mr Coyne (it was taken from the Post Office’s pleadings, but it is a theme that runs through both its submissions and also the evidence of Dr Worden) is that there are both what are called technical controls, and also operational procedures and practices conducted both by the Post Office and by SPMs, to increase the reliability of data. The passage of the Defence that was put was as follows: “Q. So robustness -- we can move on to paragraph 54 on page {C3/3/23}: " ... in addition to the technical controls referred to above, there are several operational procedures and practices conducted by Post Office and subpostmasters that serve to increase the reliability of the data stored in the central data centre as an accurate record of the transactions entered on branch terminals. Then there is a list of those. So you will see that what's being said is that the robustness of the system is said to be affected both by technical measures and by human measures. Do you see that? A. Yes, I can see that. That is what is said, yes.” 756. In my judgment it is correct to describe the second part of these, as the Post Office did in its pleading, as “human measures”
“I'm using it as a term to refer to measures and controls in existence, both technical and human measures and controls, which are designed or have the effect of identifying bugs in Horizon or identifying the consequences of bugs in Horizon so that they can be looked at and fixed.” 775. This means therefore that the Post Office was relying upon matters outside the definition of the Horizon System at the beginning of the Horizon Issues, which stated “ “The Horizon System” shall for the purposes of this list of issues mean the Horizon computer system hardware and software, communications equipment in branch and central data centres where records of transactions made in branch were processed, as defined in GPOC, at §16 and as admitted by Post Office in its Defence at §37.” 776. As Mr Coyne explained, countermeasures in the system are addressed at the design stage: “Typically countermeasures are a design aspiration. When you sit down to design a computer system and what controls are in place, then you would consider what controls are required, where to position them. Should it be the case that the controls fail do you have a human aspect doing a secondary check? It could go all the way to employing auditors to re-check figures. You could go on ad infinitum with – Q. So you accept, then, that these controls or countermeasures, whatever you would like to call them, some of them have the effect of stopping the bug in its track. Well, some of them have the effect of stopping a bug arising in the first place? A. The best way of dealing with defects is to ensure that they don't arise in the first place ……..so what you would do is throughout your testing regimes you would flush out all of the defects at that point in time, and you would put in controls to ensure that if a defect does trigger, there is a way that you are alerted to it and it is fixed as soon as possible. Q. Yes. So one of the objectives is to have a testing regime which picks up as many bugs as possible? A. Indeed. Q. But I think you accept that it is impossible to have a testing regime that picks up all of them? A. It is impossible to have a testing regime that picks up all of them and that is why you have a process of categorisation and you would allow a product to go live if -- one of the terms that's used -- all severity 1 and severity 2 defects have been dealt with. Severity 3 might be largely cosmetic and you would allow a system to go live with cosmetic defects. Q. That's not quite what I'm asking you. You accept there are going to be bugs which arise in the operation of any IT system which would simply evade any test that any human being can devise? A. There's always going to be bugs that are unknown to be within the system at the point that you go live and they are only discovered sometimes weeks afterwards, sometimes years afterwards.” 777. Both defensive programming and impact management are part of, or are descriptive terms concerning, countermeasures. In my judgment they are both more to be described as “technical controls” (to use the language of the Post Office’s pleading) rather than “human measures”
“Regarding POL-0032919.pdf, the GoldenGate replication between Oracle 10g and 11g being aborted and resulting in a number of branches reporting cash declaration and stock reporting discrepancies, were any transaction corrections sent to the 247 affected branches as a result of the discrepancies and which branches were affected by the incident?” 779. The answer that was given by the Post Office’s solicitors was as follows: “The relevance of this request is not understood. Transaction Corrections are issued by Post Office to correct transient discrepancies in branch accounts in order to restore the correct position.” 780. This answer to a request by the claimants’ IT expert therefore – correctly in my judgment – identifies that TCs are not part of the Horizon system. That is entirely at odds with them being part of a countermeasure within the Horizon system. This is effectively accepted in the Post Office’s opening submissions which expressly state at paragraph 114 that the “Post Office’s reconciliation processes are outside the scope of the Horizon Issues”
“When there is an incident involving a reconciliation exception in Network Banking which has been fully processed, then the transaction needs to be set to F99 to indicate that processing is complete. Therefore, this is done for any transaction that appears in a reconciliation report, once the resolution is complete. The reconciliation process document is SVM/SDM/PRO/0012 RECONCILIATION AND INCIDENT MANAGEMENT PROCESS and describes how reconciliation is carried out. SVM/SDM/SD/0020 END TO END RECONCILIATION REPORTING is the specifications of the reports that are produced for the reconciliation service. The CCD that describes the contracted service: SVM/SDM/SD/0015 RECONCILIATION SERVICE: SERVICE DESCRIPTION.” 784. The next request was “How often has [setting an NB102 exception to F99] occurred?” 785. Initially the Post Office would not answer this, stating inter alia that it was very unlikely that the information would have been pooled or collated. When Mr Coyne persisted, and pointed out it would be very different if the answer was “it was an isolated incident in 2003” or “it was 10,000 transactions each day for the last ten years”, an answer was provided by the Post Office through its solicitors. I shall reproduce it in full: “Fujitsu currently "F99" 10,000+ transactions per week across all NB102 associated reports (DCP and NBS)”. 786. This was correctly understood by Mr de Garr Robinson for the Post Office who (understandably) corrected Mr Coyne when he initially said he thought it was hundreds of thousands per week. “…I asked for the number of reconciliation problems that required manual intervention, and the answer I was given was that it was hundreds of thousands per week, I think was the – Q. 10,000 a week that have to be F99ed, do you remember that? A. Yes.” 787. So far as the next request, which was associated with this, is concerned, that paints the same picture of the Horizon system in my judgment. The request was: “What is the cause of an ‘Uncleared Transaction Corruptions’ and how often do [Uncleared Transaction Corruptions] occur?”
“The DCP report NB102 section 3 (Uncleared Corruptions) usually contains 100s if not 1000s of transactions per week (Kiosk transactions only). This is where there is a discrepancy on banking and payment card transactions. To determine the exact cause of any individual Uncleared Transaction Corruption, that particular occurrence would have to be analysed. However, the most usual cause is for a banking request or authorisation failing to reach its destination, which is something that can have a variety of causes. An example may be that a bank may authorise a transaction, but that confirmation is not delivered to the counter so the SPM believes that the transaction in fact failed due to some issue.” 788. This is a great number of transactions per week that require manual intervention – over 10,000, on the numbers provided by Fujitsu. I consider that Mr Coyne is correct when he says “this suggests that something is going wrong”
“There is an increased risk that unauthorised and inappropriate changes are deployed if they are not adequately authorised, tested and approved prior to migration to the production environment”
“I believe that we shall have to confess to POCL that we have not followed the correct procedure and we should advise that POCL make manual payments…..”
“1. Improvements to outsourcing application management (Quality and Change Control; managing non-functional requirements; testing good practice); 2. Improvement to segregation of duties within the “manage change” process (Quality and Change Control, Security); 3. Strengthen the change management process (Quality and Change Control); 4. Strengthen the review of privileged access (Security).” 791. Mr Coyne also relied upon the fact that although both Legacy Horizon and Horizon Online did contain a number of measures and controls, these mechanisms had been shown to have failed, and this was agreed by the experts. The minutes of the Post Office’s own Risk Compliance Committee meeting of18 September 2013 showed that not all of the risks identified in the then-recent E&Y audit had been addressed. All four pages of these minutes were heavily redacted, but one surviving entry shows that risk not addressed was “relating to the communication by Fujitsu of changes made to the Horizon system, was still outstanding. It was identified that it would cost over a£1m to implement the mitigation being suggested by the audit and that this was not proportionate to the risk being managed.”
“1.1 We agree that bugs were possible, and the table above displays a number which in the opinion of either or both experts appeared to have impacted branch accounts. 1.2 Referring to the table of bugs above, the experts agree that the bugs in rows 1, 2, 3, 10, 13, 14, 18, 23, 24, 25, 27 and 28 may have had financial impact on branch accounts. Other rows, the impact is not agreed between the experts.”
“1.15 The number of distinct bugs, for which the experts have seen strong evidence of the bug causing a lasting discrepancy in branch accounts, is between 12 and 29.” 793. Just before the short adjournment on the final day of his cross-examination, Mr Coyne was asked about the figures in that range. He was also asked to “forget Horizon Issue 4”
“Q. Are you suggesting that you have only ever suggested 21 and no one was suggesting anything above 21? A. No. There are 29 bugs, errors and defects in the list and in the table starting at number 1, but some of those go to other Horizon Issues apart from Horizon Issue 1, and they do say that clearly in the heading.” 795. There is no doubt that Mr Coyne made matters a little hard for himself during this passage of questioning, and the Post Office seized upon his change of number (which was, after all, an agreed number in an agreed statement by both experts) and made much of it: “Q. Mr Coyne, everybody on this side of the court, and I mean everybody on this side of the court, believed that your view was that the evidence showed that there were 29 bugs with lasting financial impact. You are now saying that we made a mistake, are you? A. Yes. Q. Could I ask you to explain where you made it clear what you were actually saying? A. Well, as set out in my second report, there is a table there that quite clearly says whether there is evidence of branch impact or not. And there is also a list of which Horizon Issue it specifically relates to. The only addition to that is when you come to the very last column, there's other bugs, errors and defects that are re-introduced and they are only covered in the joint statements.” 796. Mr Coyne also said: “A. Throughout the report I have set out next to every single defect whether it is relating to the test of Horizon Issue 1 or a later Horizon Issue, and I have also set out whether I believe there was impact on branch accounts all the way through report number 2. Q. Let's not talk about report number 2 because the statement -- the most up to date statement of the position, I think you agreed with me at the beginning of today, the culmination of all your work, including work you did after your second report, is the bug table in joint statement 2? A. Yes, but when you put that to me I said that that can't be read in isolation, it must be read with the second report.” 797. So far as the number is concerned, Mr Coyne settled on 22 different bugs, his explanation of his earlier different numbers was as follows: “Q. And I would like to ask you why is it you thought there were 21 bugs at 2 o'clock, whereas you thought -- was it 14 bugs at 1 o'clock? 13 bugs at 1 o'clock. A. Because I was just counting down the table at number 1 in my report. In the pressure of the situation I just tallied that column down, I didn't have enough time to consider it fully. Q. To go back to a question I asked you earlier, where do you make this clear in the joint statement? A. In the heading next to the -- in the bug, error or defect.” 798. I accept that explanation. Although Mr Coyne was criticised for certain entries in the 2 nd Joint Statement, including the headings, and it was put that the “joint statement was seriously misleading”
“A. No, I didn't because there were quite a few statements of this "in a million" nature in my report. I made careful checks of the main thread that led to my key result in section 8.7, and that was my focus. And similarly on the transaction corrections I had made checks to that spreadsheet and, you know, I included the spreadsheets with the reports so that you can see how the arithmetic is done from the spreadsheets. Q. And some of it you point out is basic arithmetic? A. I think all of it is basic arithmetic.” 807. He said, in relation to a question as to whether Mr Emery, his assistant, had “checked the maths” that “No, he didn't actually. The maths is there in the spreadsheets for anybody to check. It is not advanced statistics, it is basic multiplication and division, and it is laid out in the spreadsheets.”
“617. I shall assess the financial impact of bugs in Horizon over three different scopes: 617.1. (a) Across all Post Office branches, during the lifetime of Horizon. 617.2. (b) Across all Claimant branches, while they held them. 617.3. (c) On a single Claimant branch in a single month. 618. It is possible to relate the financial impacts on these scopes, by numerical scaling factors. I calculate those scaling factors in this sub-section. 619. Over the period 2000 – 2018, (i.e. 19 years) the Post Office network has consisted of more than 11,000 branches. The mean number of branches in all years over the period has been about 13,560. This figure is derived from the spreadsheet referred to at paragraph 178 of Ms Van Den Bogerd's Second Witness Statement, assuming that the spreadsheet is accepted. If this evidence is accepted, the number of 'branch months' (a single branch, trading for a single month) has been 13,560 * 12 * 19 = 3,091,680. This is the number of monthly branch accounts that have been produced. 620. This means that for a typical Post Office branch, the scaling factor between scope (a) and scope (c) for the impact of bugs in Horizon is a factor of 3 million. 621. For Claimants' branches, rather than typical branches, the scaling factor of approximately 3 million may need to be adjusted for two possible effects: 621.1. It might be asserted that Claimants' branches are more or less likely than other branches to be hit by bugs in Horizon, because of some special property of Claimants. 621.2. Claimants' branches may, on average, be smaller or larger than typical branches across Post Office network. If they are smaller, they handle fewer transactions in a month, and so are less prone to Horizon bugs in those transactions. 622. It seems implausible to me that there is some special factor about Claimants' branches, which makes them much more prone to bugs in Horizon - bugs which one would expect to strike any branch at random. Nevertheless, I have considered the possibility carefully in Appendix F. I have shown there that there is no significant difference between Claimants' branches and other branches, in proneness to bugs in Horizon. 623. It appears, from the spreadsheet attached at paragraph 179 of Ms Van Den Bogerd's Second Witness Statement, that the Claimants' branches are, in terms of customer transactions carried out per day, smaller than the average across the whole Post Office branch network. 624. If this spreadsheet is accepted, it implies the following about Claimants' branches: 624.1. From summing rows of the spreadsheet, the 561 Claimants' branches carried out 558,000 customer transactions per week in 2007. 624.2. This is 558,000/6 = 93,000 transactions per day, assuming a Post Office branch is open for 6 days a week. 624.3. Across 561 Claimant branches, this is an average of 93,000/561 = 165 customer transactions per branch per day. 625. For comparison with this figure, I need to estimate the average size of branches across Post Office Network. I have done this using two pieces of evidence. Ms Van Den Bogerd's Second Witness Statement says that, across the whole Post Office network, there are approximately 48 million customer transactions per week, or 8 million per day in 2017 (assuming again that branches are open for 6 days in the week). [626. I have omitted the table of EPOSS Transaction Rates, which are then summarised in 627] 627. The best estimate I can make of the average daily volume in 2003 from this table is to divide the peak month by 26 working days. This gives an estimate of approximately 4 million transactions per day in 2003, compared with 8 million transaction per day in 2017. 628. I therefore estimate that the average volume of transactions over the period 2000-2018 has been approximately 6 million transactions per day, mid-way between the 2003 figure and the 2017 figure (assuming Mrs Van Den Bogerd's evidence is accepted. I note that Mr Coyne quotes the same figure from her). 629. To estimate how much smaller the typical Claimant's branch is than the average Post Office branch: 629.1. The average transaction rate for all Post Office branches is 6,000,000/13,650 = 439 customer transactions per branch per day. 629.2. So, in terms of customer transactions per day, the typical Claimant branch was smaller than the average Post Office branch by a factor 165/439 = 0.37. 630. So, Claimants' branches, being generally smaller than Post Office average, have fewer transactions per month and so are less likely to be hit by a Horizon bug in a given month. (The calculation of the factor 0.37 contains some minor approximations that can be improved with further effort. I intend to do so in my supplemental report.) The factor 0.37 increases the scaling factor above, between scopes (a) (see paragraph 617.1) and (c) (617.3) from about 3 million to about 8 million. 631. I illustrate what the factor of 8 million means using a hypothetical example of a bug which has occurred 16 times over the lifetime of Horizon, with mean financial impact on these occasions of£1000 . Call this Bug A. The financial impact of Bug A is similar to that of the Suspense Account bug. 632. If I selected a Claimant's branch and a month at random, then the chances of Bug A occurring at that branch in that month are only 16 in 8 million, or 2 in a million - an extremely small probability. 633. Different types of bug occur independently of one another, so their probabilities of occurring are additive. If there were a second bug similar to the hypothetical bug above - call it Bug B - then the chances of either Bug A or Bug B happening to one branch on one month are twice the previous figure - 32 parts in 8 million. 634. If there were 100 similar bugs - called Bug A, Bug B, Bug C,... Bug Zz - the chances of any one of them happening to one branch on one month are still only 100 * 16 parts in 8 million, or one part in 5,000. This is still a very small probability. 635. It then follows that in order for one occurrence of a bug, of similar financial impact to the Suspense Account bug, to have even a one-in-ten chance of occurring to one branch on one month, there would need to be 50,000 such distinct bugs - because 50,000 * 16 / 8,000,000 = 800,000/8,000,000 = 1/10. There would have to be a Bug A, Bug B, Bug C, and so on, in a list with 50,000 distinct bugs. 636. Even if there were some 'super-bug' - with financial impact ten times larger than the suspense account bug - there would have to be approximately 5,000 such super-bugs to give a one in ten chance of affecting a Claimant's branch accounts in a given month. There is no evidence for even one such super-bug - let alone for 5,000 different ones.” [637 and 638 then provided a further table and calculation that I omit] 639. The Claimants have never asserted that there are as many as 50,000 distinct bugs in Horizon, with each bug on the same scale of financial impact as the Suspense Account bug. Their case rests on two or three known bugs of this scale, and on the unproven assertion that there may be others. 640. If any Claimant were to assert that, for instance, a deficit of£1000 had occurred in his branch in a particular month, caused by a bug similar to the Suspense Account bug, the chances of that assertion being correct are extremely small, because Horizon bugs strike so rarely - unless Horizon contained of the order of 50,000 distinct bugs of that kind (and even then, the chances are only one in ten). 641. Mr Coyne has examined more than 5,000 KELs, and not found definite evidence for even one bug with impact similar to the Suspense Account bug - let alone 50,000 of them. 642. The implication of this result - the very small probability of any error in one months' accounts from a bug in Horizon - is that the accounts for any branch on any month are overwhelmingly likely to be correct (apart from effects such as delayed TCs, which are corrected after a variable delay). 643. In my experience, no commercial IT system could ever go live with as many as 50,000 serious bugs - and certainly could not have the good in-service record over 18 years that Horizon has had.”
“38. The Defendant operated one or more suspense accounts in which it held unattributed surpluses including those generated from branch accounts. After a period of 3 years, such unattributed surpluses were credited to the Defendant's profits and reflected in its profit and loss accounts. 39. The Defendant thereby stood to benefit and/or did benefit from apparent shortfalls wrongly attributed to the Claimants which did not represent real losses to the Defendant.” 811. This was pleaded to by the Post Office in the Defence as follows: “73. As to paragraph 38: (1) Post Office does not have a clear understanding of what the Claimants mean by the term "suspense account" or by the phrase "unattributed surpluses including those generated from branch accounts". (2) In relation to its dealing with its clients, Post Office is almost always able to reconcile its figures to those of its clients. However, as is inevitable given the nature and scale of those dealings, occasionally Post Office is not able to do so. For example: (a) A client, such as a bank, makes a payment to Post Office in respect of a particular transaction undertaken in a branch but the payment exceeds the amount that Post Office considers to be due. (b) Post Office seeks but does not reach agreement with the client as to the amount due and the client does not accept repayment of what Post Office considers to be the overpayment. In parallel, Post Office also seeks to determine whether the overpayment should be credited to a branch or to any other relevant part of its business. (c) Where the proper beneficiary of a credit cannot be determined, Post Office temporarily records the overpayment in its accounts. If the overpayment is not resolved within 3 years, the overpayment may be credited to Post Office's profit and loss account. (3) The existence of processes for recording and resolving credits of the type described above are an ordinary business practice.” 812. This was then pleaded to by the claimants in their Reply and Defence to Counterclaim: “29. The Defendant's professed lack of clear understanding of the term 'suspense account' at paragraph 73(1), is noted at paragraph 2.1 above, and its response to the Claimant's RFI is similarly evasive. 30. The effect of the Defendant's example pleaded at paragraph 73(2), is that the Defendant admits the possibility that: 30.1. a sum of money may be received which the Defendant believes to be an 'overpayment' from a third party client (e.g. a bank) in respect of a transaction undertaken in a branch: paragraph 73(a); 30.2. the Defendant would seek to determine whether the overpayment should be credited to a branch, or to any other relevant part of its business: paragraph 73(b); 30.3 where the Defendant does not identify ‘the proper beneficiary’ of the credit, Post Office ‘temporarily’ records the overpayment in its accounts and credits that to its profits, if not resolved within 3 years. 31. This logic would apply where the Defendant’s internal records were wrong due to a Horizon error or in the case of any failure to identify the relevant branch or Subpostmaster”. 813. I find that there is no sensible basis for the professed lack of understanding by the Post Office in its Defence of the term “suspense account”
“2.11 We note that Post Office's control and reconciliation procedures rely on correct information being supplied by third party clients. It follows that, if incorrect information is provided by any client company, this can give rise to a loss being charged to a branch. We also note that, for most of the past five years, substantial credits have been made to Post Office’s Profit and Loss Account as a result of unreconciled balances held by Post Office in its Suspense Account.” 816. The question and answer are as follows, having read out the passage: “Now I'm not asking you about that as evidence of itself, I'm just asking were you aware that that was a concern that Second Sight had raised about substantial credits being made to Post Office's profit and loss account as a result of unreconciled balances held by Post Office in its suspense accounts? A. I read the Second Sight reports and the Post Office's reply. Q. But did you spot that? A. I didn't spot that, no.” 817. This point was put in the context of evidence Dr Worden had given, namely the effect of small bugs or what he called “micro bugs” being unlikely to have an effect. “Q. Okay. Well, what about -- because elsewhere in your report, I don't want to stray too much at the moment, but you also mention micro bugs, small bugs, being unlikely because you would expect to see large sums of money somewhere -- A. Money that disappears somewhere has to pop up -- Q. It's got to come back somewhere else, it's got to pop up somewhere else.” 818. The above passage was then put to him. Dr Worden’s failure to spot what is, in my judgment, a point directly relevant to the likelihood of Horizon bugs, errors and defects having a potential impact upon branch accounts is a notable one. Whether the point is taken from the pleadings, or from the Second Sight report (which Dr Worden accepted he had read, together with the Post Office’s reply to that document) does not much matter, in my judgment. I consider that this one omission alone - which I consider to be a serious omission on his part – is sufficiently fundamental as to undermine his evidence on this issue on its own. 819. The following points are relevant in respect of Dr Worden’s statistical exercise: 1. Statistics is the branch of mathematics that concerns the collection, presentation and analysis of data. There was no permission granted by the court for expert statistical evidence in this case. The permission that was granted was for expert IT evidence. 2. Had either party sought permission to rely upon evidence from an expert statistician, I would not have granted it. Statistical analysis does not assist in resolution of the Horizon Issues, which required (in my judgment) detailed technical analysis and consideration by IT experts of the detailed occurrences of what could, potentially, be bugs, errors and defects. This was so that their expert opinion could be brought to bear, firstly, upon the important issue of whether there were any, and also if there were, their extent and the degree to which they potentially could impact branch accounts. This both experts did in any event, and that evidence – based on contemporaneous records and evidence about their occurrence - is of far greater assistance than this wholly theoretical argument about why Horizon bugs cannot have affected the claimants’ branch accounts. 3. Dr Worden uses statistics, as he uses mathematics, as part of his professional life. Any particle physicist would, and does, use mathematics every day. Dr Worden accepted that this was not advanced statistics, and he was right to do so. It is a combination of simple assumption together with basic multiplication and division. The assumptions are not, in my judgment, valid. 4. None of the Horizon Issues required the experts to perform a mathematical or statistical analysis of the likelihood of the claimants’ alleged shortfalls and discrepancies being caused by bugs, errors and defects in Horizon. Horizon Issue 3 – using as it does the term “extremely unlikely to be the cause of shortfalls in branches” – does make this exercise by Dr Worden relevant if that was how he chose to address that issue. That does not however make it either correct, or of any assistance. 5. The accepted correction to the figure of “2 in 10 million” – which he accepted before his cross-examination started should be “2 in a million” – shows, in my judgment, how very broad brush this exercise was. As he put it in his oral evidence, “…there were quite a few statements of this "in a million" nature in my report”. 6. It is plain that Dr Worden’s own evidence accepts that there are two fundamental assumptions that are central to this exercise. The first is that any bugs in Horizon strike branches randomly (which is spelt out in his report). The second is that the claimants’ branches are a random sample. This can be seen, implicitly, from his statement at paragraph 621.1 that “it might be asserted that Claimants' branches are more or less likely than other branches to be hit by bugs in Horizon, because of some special property of Claimants” and also at paragraph 632 “if I selected a Claimant's branch and a month at random….”
“632. If I selected a Claimant's branch and a month at random, then the chances of Bug A occurring at that branch in that month are only 16 in 8 million, or 2 in a million - an extremely small probability. 633. Different types of bug occur independently of one another, so their probabilities of occurring are additive . If there were a second bug similar to the hypothetical bug above - call it Bug B - then the chances of either Bug A or Bug B happening to one branch on one month are twice the previous figure - 32 parts in 8 million.”
“A. Well, my analysis of the claimants' evidence is mainly contained in my supplemental report, and I explained there that individual claimants' evidence, particularly individual subpostmaster evidence, I did not feel able to make strong use of that and I gave the reasons in my supplemental report. So my opinions have little dependence on that. And the core of my opinions, the numerical estimates I make, those estimates have been designed -- or the process and the method has been designed so that if -- so that to make my assumptions evident where those assumptions come in, and if the court finds something different from my assumptions the court can go to the spreadsheets and re-do the method for itself.”
“Q. Have you had any changes of heart about anything in your report having heard it? A. Changes of heart? Q. Anything you want to change about what you said? A. Well, had there been some fundamental change I would have felt obliged to communicate it to the court. Q. Of course, because you take your duty very seriously? A. Yes, I do. I can't think of any major change of heart, but there may well be things that you bring me to and I say there's an adjustment here. But to come back to the point about precision, I am conscious of how precise my numerical results have to be in order to be of assistance to the court.” 839. This effectively must mean that Dr Worden did not consider that any of the changes, in the Fujitsu evidence in particular from the cross-examination, would impact upon his expert opinion. This is notwithstanding that he had assumed that the Post Office’s evidence would all be accepted, and anyone present for Mr Godeseth’s cross-examination can only have concluded that the factual position presented by Fujitsu in the written witness statements was very far from an accurate one. Indeed, the extent to which the cross-examination did not have any effect – either appreciable or otherwise – on Dr Worden’s expert opinions shows how very theoretical the whole exercise was. 840. Finally on this point, due to the disruption caused by the recusal application, and the unavoidable re-arrangement of the trial timetable, the experts came to give their evidence in June 2019, and all of the factual witnesses who were cross-examined (with the sole exception of Mr Parker) had completed their evidence by21 March 2019 , many weeks earlier. Even Mr Parker had completed his evidence by11 April 2019 . Dr Worden was called on11 June 2019 , two months after this. It is unusual for there to be such an interval in a trial, but circumstances in this case led to extra time being available. There was more than ample time for Dr Worden to have identified how, if at all, the Post Office’s evidence of fact that emerged earlier in the trial (in particular from Mr Godeseth, but also from Mr Parker too) impacted upon his opinions. It plainly did not. 841. I will provide only one example of Dr Worden’s unsatisfactory approach to the evidence, although there were many. In terms of whether transactions could be inserted at the counter in Legacy Horizon, what is called “remote access”, Dr Worden had in his first report concluded that Mr Roll (a witness of fact) was factually wrong. He expressly stated that he, Dr Worden, had “established” that Mr Roll was wrong. He accepted that use of the word was wrong but what he had done in his 2 nd report was equally telling, in my judgment. This was after those witnesses who were to be called for the Post Office, still employed at Fujtisu, had corrected (or partially corrected) their erroneous first witness statements, and accepted that remote access without the knowledge of the SPM was possible. He said (at paragraph 83) that he required: “further factual information before I can comment on this evidence. Which 'specific person'? Under what circumstances? How frequently? Until I have that information, it remains possible in my view that any transaction which 'would appear to the subpostmaster as though it had been carried out through the counter in branch' might only be a transaction that he had given his consent for, as the 'specific person' - and which had in effect been made on his behalf.” 842. Dr Worden also then stated: “84. Therefore, Mr Roll's new evidence does not cause me to alter the opinion expressed at paragraph 1119 of my first report, when commenting on Mr Roll's first witness statement, that he could not alter branch accounts without the Subpostmaster knowing.” 843. This approach by Dr Worden to the evidence for the claimants on the one hand (Mr Roll) and the Post Office on the other (Mr Parker and Mr Godeseth in their statements) was explored in cross-examination: “Q. Yes, the one that we agreed was incorrect in approach and conclusion. A. We agree now, but at the time I wrote the second report, on the basis of Mr Roll's evidence and Mr Parker's evidence, I didn't see reason to change that opinion. Q. I would suggest to you that you were inexplicably reticent to accept something that was contrary to Post Office's interests. A. No, I was reticent. Not inexplicable, I was reticent because I had not seen sufficient evidence to convince me that these things could be done without the knowledge of the subpostmaster . Q. But Dr Worden, would you accept that the approach you have taken there contrasts very, very strikingly in how you approach Mr Roll's evidence with the approach you took at 1119 in your first report when you accepted, effectively, Mr Godeseth's evidence? A. I have accepted that my approach at 1119, that use of the word "established" was wrong and my approach was wrong, and we have established that -- you know we have done that before the interval – Q. My question is: do you accept the contrast is very striking? A. I think the court will have to -- I accept that my attempt to make my position clear that I'm not trying to find findings of fact, I'm not trying to find one witness or the other, I accept that on this occasion I fell short of that . Q. Do you mean only in your first report or also in your second report? A. Not in the second report . I believe the second report was -- you know, I believe that the evidence from Mr Parker and Mr Roll, and if we look again at Mr Roll's paragraph 20, I believe that the question that I asked in paragraph 83 that I want further evidence, I believe that was a valid approach . Q. When you say in paragraph 84 -- because we have dealt with what you accept about your first report at paragraph 1119 several times, haven't we? A. Yes. Q. So in your second report, you say, notwithstanding what you have now accepted about 1119, in your second report that you were right not to change that view. A. I hadn't seen sufficient evidence to change that view . Q. But do you say you were right not to change that view when you revisited the entire piece in your second – A. At the time I believe it was right not to change that view because I hadn't seen evidence that convinced me that this change could be made without the knowledge of the subpostmaster. That's where I am . Q. Dr Worden, do you accept that that betrays a complete failure to appreciate the need to consider the situation both on the basis of whether the claimants' evidence is right as well as on the basis that the defendant's evidence is right? A. No, I don't accept that.”
“the Horizon System” shall for the purposes of this list of issues mean the Horizon computer system hardware and software, communications equipment in branch and central data centres where records of transactions made in branch were processed”. 848. However, it can be seen simply from the descriptions above, which are Dr Worden’s own descriptions, that not all of the 18 are even part of the Horizon system. Some are simply manual checks. Some genuinely are countermeasures, but Dr Worden groups both types together. 849. This can be seen from the following. Transactional Integrity and database recovery or TIN is undoubtedly a countermeasure within the Horizon system. The explanation given by Dr Worden in his summary table for this is “database management systems provide many facilities so that numerous kinds of failure cannot leave the data in an inconsistent, unusable state, or lose any data that have been previously stored.”
“Recovery in a database system means, primarily, recovering the database itself: that is, restoring the database to a correct state after some failure has rendered the current state incorrect, or at least suspect. (We will elaborate on what we mean by "a correct state of the database" in the next section.) And the underlying principle on which such recovery is based is quite simple, and can be summed up in one word: redundancy. (Redundancy, that is, at the physical level; we do not want such redundancy to show through to the logical level, naturally, for reasons discussed in detail elsewhere in this book.) In other words, the way to make sure the database is recoverable is to make sure that any piece of information it contains can be reconstructed from some other information stored, redundantly, somewhere else in the system. Before going any further, we should make it clear that the ideas of recovery (indeed, the ideas of transaction processing in general) are largely independent of whether the underlying system is relational or otherwise—though it is true that most of the theoretical work on transaction processing has historically been done, and continues to be done, in a relational context specifically.” 851. This is a countermeasure. Transactional integrity is intended to give protection against a wide variety of conditions, such as hardware errors or communication failures, software errors (which can lead to failures and cancellations) and operations being cancelled when they have not been completed. Such cancellations can be made by what Dr Worden calls the operator (the SPM or their assistant) but cancellations can – and the factual evidence shows this occurred – be caused by outages, power cuts and other incidents. A lay person might say that a system has “crashed", or one of the terms used in the factual evidence in this trial was “system outage”. 852. Other measures are not strictly speaking countermeasures, they are sensible design intentions (Mr Coyne used the term design aspirations). An example of this is defensive programming or DEP. Since programming is at the heart of any software – indeed, programming creates the software itself – the way that programs are originally written (what could be called the programming of the system) constitutes the very system itself. DEP was described by Dr Worden in the following way in his table summary: “software is divided into small self-contained modules, which do not assume that other modules are correct, but defend themselves by checking their inputs and raising alerts early.”
“There are many differences of detail in how reconciliation is carried out for different client organisations, or where it is carried out; sometimes the client organisation does it from a file sent to it by Post Office, and sometimes Post Office does it. However it is done, and wherever it is done, the result of reconciliation is always in principle the same.” 856. The way that Dr Worden explained the process was as follows: “285. When reconciliation finds a transaction for which Post Office's record and the client record do not match, it is passed to a department in Post Office accounts which handles reconciliation discrepancies. Each such discrepancy is, until it has been dealt with, an error in the accounts - and so it must be dealt with. The task of this department is to determine how each discrepancy arose , which therefore determines how it needs to be handled. 286. When the appropriate department in Post Office decides that responsibility for a discrepancy lies with the branch, a request for a TC is issued and is passed from POL FS to the BRDB in Horizon Online. At this point, there is no impact on branch accounts. The request is passed on from the BRDB to the branch Horizon system, so that it will show on the Subpostmaster's screen when he starts the Horizon system the next morning. At this point, the Subpostmaster may either accept the correction or may question it and ask for further investigation.”
“5.1 Reconciliation between transactions recorded on Horizon and transactions recorded by Post Office's clients is largely automated. Detected discrepancies were subject to manual corrective fixes and/or the issue of Transaction Corrections/Error Notices to the Subpostmasters.” and: “5.3 The adequacy of Post Office back office processes to prevent discrepancies in branch accounts can be measured by the quality of the TC process. This quality includes: • The processes of consideration of available data • The level of errors observed in the process • The level of complaints or disputes raised following a TC • The level of upheld complaints following a TC • The level of financial impact of erroneous TCs.” 3. Under sub-topic Third Party Data: “5.4 Errors in third-party data have led to discrepancies in branch accounts, through erroneous TCs being issued on Subpostmasters. 5.5a PO does not control the level of errors made by its third-party client organisations (which may lead to errors in TCs), or the delays in their processes (which may lead to delays in TCs). 5.5b PO can and should ensure, by careful investigation of disputed TCs, that only a small proportion of errors by PO clients lead to losses for Subpostmasters, provided that the Subpostmasters are in good control of their branches and have the required information available.” 859. This shows that not only do the experts treat TCs as being outside Horizon in the Joint Statement, and also agree that the SPMs must have available the required information, but also agree that the level of upheld complaints in TCs (and the value of erroneously issued ones) is an ingredient required in order to measure the quality of the process. I find that the level of upheld complaints in TCs, and the value of erroneously issued TCs, is necessary in order to assess the quality of the Post Office back office processes to deal with discrepancies. That data was simply not kept, according to the Post Office witnesses. Further, Mr Coyne’s analysis showed that 77% of disputed Santander TCS were upheld. Entry 5.6 in this Joint Statement, not agreed by Mr Coyne, was an entry by Dr Worden which stated: “The figure quoted by Mr Coyne (77% of disputed Santander TCs upheld) illustrates that this process worked well.” 860. This means that Dr Worden’s analysis of “working well” was 77% of TCs issued in respect of Santander transactions which were challenged by SPMs was upheld. It is obvious that if 77% upheld, this means that 23% of TCs were not upheld, in other words were TCs that should not have been issued. I do not consider that a ratio of approximately 1 in 4 TCs being incorrectly issued can be properly described as the process “working well”, particularly as each TC would have a direct impact upon branch accounts. In my judgment, for the process of issuing TCs to be “working well”, the level of accuracy required would be far higher than 77%, and in excess of 90% (if not in the high 90s as a percentage) at the very least. Nor do I consider that to be unrealistically or unachievably high, given TCs impact upon branch accounts, the accuracy of which is required to be very high, and expected by the Post Office to be such that shortfalls in branch accounts have to be paid by the SPMs. 861. As put to Dr Worden in his cross-examination, if his approach to interventions by human beings were to be extended to its logical conclusion, the group litigation itself would be categorised as a Horizon countermeasure. In my judgment, that would be the logical consequence of his approach. Plainly this litigation cannot properly be seen as a countermeasure. His approach to countermeasures is not a sensible one; he mixed up both genuine countermeasures such as TIN; design functions such as DEP; and manual corrections as well, and treated them all in the same way. He was wrong to do so. 862. It was also somewhat unhelpful for some of his countermeasures, such as later correction of user errors or UEC, to be so widely defined as to include both manual checks – including by the SPM themselves – and the Post Office and Fujitsu’s both automated and manual processes. Another criticism of Dr Worden’s approach to countermeasures relates to one which Dr Worden described in his paragraph 463 as being “one of the most important countermeasures in Horizon”, and that is manual inspection of data. I do not see how this can be a countermeasure in Horizon as defined at all. Dr Worden’s summary states that “Any large business IT system is used by many people, who view its outputs and check them against each other for consistency, and against their own knowledge of the business. Subpostmasters, watching their branch accounts, were a key component of this .”
“His witness statement describes several different audit processes, designed to ensure by independent external review that quality is managed effectively in Horizon (Witness Statement of Mr William Membery,28 September 2018 ). If these audits have taken place as described in his witness statement, and if the results are broadly as he describes them, then that would increase my confidence that the quality of Horizon has been effectively managed.” 866. He also stated that “a large system such as Horizon can only be robust if sufficient attention has been paid to issues of quality in its development, testing and support.”
“I understand that my duty in providing this report and giving evidence is to help the Court on matters within my expertise, and this duty overrides any obligation to the party by whom I am engaged or the person who has paid or is liable to pay me. I confirm that I have complied and will continue to comply with this duty. I have not assumed that any particular version of events is true and I have had regard to the case of Imperial Chemicals Ltd v Merit Merrell Technology Ltd[2017] EWHC 1763 (TCC) in producing my report”. 877. That particular case (of which there are a number of judgments, on liability and quantum) concerned an expert and his preference for the evidence of fact of the party who had instructed him. I had stated in that judgment at [74] that “An expert's role is not to decide issues of fact themselves, and choose what facts to believe and what not to believe”
“1. Experts of like discipline should have access to the same material. No party should provide its own independent expert with material which is not made available to his or her opposite number. 2. Where there is an issue, or are issues, of fact which are relevant to the opinion of an independent expert on any particular matter upon which they will be giving their opinion, it is not the place of an independent expert to identify which version of the facts they prefer.” 879. Dr Worden went so far as to state, in respect of his consideration of the variations in robustness of Horizon over time, that “in my opinion, as expressed in section 8, the Claimants’ shortfalls are not caused by bugs in Horizon, or any lack of robustness in Horizon.”
“Q. Sorry, not having supported the very many requests he had to make? A. Yes, I felt his interests were different from mine and I really had plenty to do. Q. And you initially only looked at KELs, didn't you, you didn't actually look at PEAKs? A. No, I looked at KELs and I looked at PEAKs where they were relevant. I felt that KELs were a more distilled form of information. I felt they were sufficient in many ways, especially when you go and look at the PEAKs. Q. You looked at one or two PEAKs referred to in the KELs you looked at? A. Yes. Look at some of them where you feel the KEL doesn't tell you enough. Q. And Mr Coyne was looking at PEAKs and KELs? A. I think in the first report he was looking mainly at KELs, like me, and in his second report he turned on PEAKs.” 887. The reason that PEAKs are of great assistance is that there is a great amount of further information contained in them than in the KELs, and they also give a far more complete picture of the number of different occurrences of a bug. Both types of documents are however useful. I do not know why Dr Worden should have seen Mr Coyne as having “different interests” to him. 888. Dr Worden also entirely seemed either to misunderstand (or restrict his consideration of) Horizon Issue 1, which concerned bugs, errors and defects having the potential to cause apparent/alleged discrepancies or shortfalls in branch accounts or transactions, or undermine the reliability of Horizon accurately to process and to record transactions. When challenged about his approach to this, including the absence of the word “cause” and the use of the word “potential” in the issue itself, he stated the following: “A. What's the distinction between having the potential and actually doing? I mean, if it doesn't actually do, then in some sense it didn't have the potential.” 889. This is not the correct, or conventional, use of the word “potential”
“We respectfully will invite the court to adopt an approach more consistent with Mr Coyne's approach which is to look at what actually happened with particular examples and trace them through to a reasoned and careful conclusion and then from the ground up, as it were, draw inferences upwards rather than from an overarching hypothesis downwards.” 891. The “overarching hypothesis” was that of Dr Worden. Because he had taken such an overall view, and an overall view on the most recent performance and functionality of the Horizon System (which is now HNG-A as of February 2017, and is no longer HNG-X), the degree of assistance his evidence provided to the Horizon Issues across the whole life of the system, both Legacy Horizon and Horizon Online, was more limited than that of Mr Coyne. 892. In this group litigation, when the first individual claims are being tried out, further analysis will be required of the actual branch accounts of individual claimants. That stage of the litigation will deal with actual branch losses, if any, that are said by the claimants to be due to the actual impact of the bugs, errors and defects which exist. It is at that stage that findings will be made about actual impact upon the branch accounts of the different claimants. Some of the bugs and defects are agreed by Dr Worden and Mr Coyne, and those that are not agreed are resolved by me in the Technical Appendix. It is against the background of those findings that individual claims will be resolved. The purpose of the Horizon Issues trial was to resolve the generic issues, as the parties were poles apart about even the existence of such bugs, errors and defects, and whether they had the potential to act upon branch accounts and transactions in the way alleged by the claimants. Dr Worden moved away from consideration of those specifically drafted and agreed Horizon Issues and answered rather different ones. 893. I also consider that Dr Worden, in his approach to countermeasures, insufficiently differentiated between measures which were part of the Horizon system and those that were not. He also included items in his countermeasures that plainly ought not to have been included as such, for example those the result of human intervention or involvement, and manual inspection of data by SPMs. 894. In my judgment, Mr Coyne emerged more or less unscathed from his four days of cross-examination by leading counsel for the Post Office. That is not to criticise the cross-examination itself, which was very thorough and careful. It is just that what the Post Office set out to demonstrate, including lack of impartiality on Mr Coyne’s part (if not positive bias), lack of reliability in his conclusions, and the other detailed criticisms of him, simply was not achieved. He was described as “evasive” in some of the Post Office’s closing submissions, a term which I find is somewhat wide of the mark in describing how he gave evidence in the case before me, and how he answered questions. He was also criticised in the same submissions as adopting “an approach [that] was simply to look for bugs without giving a proper sense of scale, context or even balance”. 895. In the Post Office’s closing submissions, there is a section headed “Judicial criticisms of Mr Coyne’s approach” where reliance is placed on a judgment in Scotland, called CGI IT UK Ltd v Agilisys[2018] CSOH 112 by Lord Bannatyne. In that case, Mr Coyne had appeared as an expert and he was criticised by the judge for an unbalanced view, amongst other things. The judgment was handed down on4 December 2018 . The fact that he had been judicially criticised in that case was not specifically put to Mr Coyne. However, the substantive criticisms of him, that he gave unbalanced and selective evidence, and the other criticisms, were put to him, and he did not accept them. The point that he had been previously criticised in a judgment ought specifically to have been put by reference to that case, so that he could answer this point. However, in any event I have read (as invited in the Post Office’s submissions) the entire Scottish judgment, in which the evidence of Dr Hunt, Mr Coyne’s opposite number, was preferred to his evidence. I have done this with particular care, and I have also considered all the criticisms made of him in that judgment, as well as the ones put to him in his cross-examination before me. The Scottish judgment provides a full explanation of the shortcomings of his approach in that case. I have also taken this prior judicial criticism into account in weighing up his expert evidence in this trial. 896. The Post Office, amongst other things stated that “it is submitted that the examples set out above [ie in the Lord Bannatyne judgment] show the same tendency to reverse and switch in response to difficult points.”
“It is a central principle of Horizon that the Core Audit Database acts as a secure 'gold standard' for branch accounts (countermeasure SEK) and that the audit record can only contain events which originated at the counter - either in customer transactions or monthly balancing.” 913. That is an accurate description in my judgment. I can see no sensible or justifiable reason for the Post Office’s reluctance to consult the audit data in cases of serious dispute with SPMs, in particular the types of dispute which form the subject of this case, and without doubt, any dispute that involves criminal proceedings against SPMs. I also consider the same point applies in relation to internal Post Office proceedings that lead to the suspension or termination of SPMs. Acceptance of that point does not mean that that the audit data has to be consulted for every transaction correction issued by the Post Office. As can be seen from the passage of cross-examination at [908] above, the Post Office effectively jumped to the issue of whether audit data should be consulted before issuing every single TC. Mr Coyne accepted it was not necessary to do that. That is not the same as accepting it should not be consulted in circumstances where there was a dispute between the Post Office and SPMs. 914. Mr Coyne had identified issues with using Credence data. There was a one-hour difference in the time stamps used between Fujitsu and Credence, which can hardly have helped sensible investigations when SPMs raised queries, but there is more to this than that. The E&Y review in March 2011 identified various issues with Credence, including weak change controls within the back-end of the systems which allowed Logica developers (the third-party provider) to move their own uncontrolled changes into the production environment, which included both Credence software code and the data within Credence used for what was called “audit evidence” but which has to be differentiated from what I am referring to audit records in the audit store. There was a lack of further documentation to approve fixes and patches applied to Credence outside of the release process, which meant that linking changes to issue tickets to record the original request for the bug fix was not possible. 915. A concern was also noted by E&Y in respect of front end change process weakness. The passage states: “During our walkthrough of user administration of the front end of Credence we noted several users with administrator rights, including some generic users (this is noted below as a separate point). These users have the access rights to create and amend reports, including those which may be relied upon for audit evidence. These users can change report design, and processing without documented request, test or approval. When users have the rights to change reports that are used by the business for reconciliation, exception reporting or other processing, there is the risk that the reports are manipulated either intentionally or accidentally.” 916. There is also at least one specific occasion, considered in the evidence during the trial, which shows that the Credence data does not show the correct position. This was put to Ms Van Den Bogerd. This was an occurrence at Lepton in October 2012. It led to something that was referred to throughout the trial as the Helen Rose Report or Rose Report, as the author of the report into it was called Helen Rose. I have dealt with it to a degree at [227] above. That report records that: “A transaction took place at Lepton SPSO 191320 on the04/10/2012 at 10:42 for a British Telecom bill payment for£76.09 ; this was paid for by a Lloyds TSB cash withdrawal for£80.00 and change give for£3.91 . At 10:37 on the same day the British Telecom bill payment was reversed out to cash settlement. The branch was issued with a Transaction Correction for£76.09 , which they duly settled; however the postmaster denied reversing this transaction and involved a Forensic Accountant as he believed his reputation was in doubt.” (“duly settled” means the SPM paid the Post Office that sum). The Credence data showed that the SPM had reversed the transaction. By consulting the audit data, Mr Jenkins discovered that he had not. This was expressly confirmed, both in the Rose Report and also by Ms Van Den Bogerd in her cross-examination. 917. This shows that the management data is not entirely reliable. Had the SPM at Lepton simply paid the approximately£80 and not engaged the forensic accountant – in other words, had he not been prepared to pursue this matter – the Post Office would simply have received that sum from the SPM to which it was not entitled, and the Rose Report would not have emerged. 918. The issue of remote access also further complicates (or rather, weakens) the reliability of management data. All of these concerns reinforce the view that I have in any event, that audit data (as in, that stored in the audit store) is what should be used in the circumstances that I have identified. 919. One point that occupied the parties a great deal in the Horizon Issues trial, upon which there was some evidence, and upon which Mr Coyne formed a view based on his analysis of some contemporaneous documents, is that Fujitsu raised a charge to the Post Office for audit data requests (also called “ARQ requests”) above a certain number per year. The numbers that were discussed were in the amount of hundreds of pounds per request. The parties could not agree on what these charges were, although it was accepted that there were a number each year that were either free, or the costs of which were included in the contract sum (depending upon which way one looks at this matter). The charging structure will, in terms of the amount per request above a particular number, have changed over the years in any event. 920. That charges are raised by Fujitsu to the Post Office is not an adequate answer, in my judgment, to the Post Office’s failure to consult or provide the audit data in cases such as those in [912] above. There are some contemporaneous references within Post Office documents suggesting this may have been a disincentive in some cases to raising ARQ requests of Fujitsu. Further, there are numerous references within Post Office documents (at a high level) of the very great cost to the Post Office of the Horizon system generally, that cost being paid to Fujitsu. Fujitsu were said in one document to see the contract with the Post Office as a “cash cow”
“RISKS • Need to be clear for the Board what operational risks we are facing and how we are mitigating those risks, are there any choices we can make to accelerate the solution to get inside the risk appetite. • Already making these changes to the areas of risk, but will make this clearer. Need a strategic discussion at GE on choices in the IT strategy • Horizon - cannot accelerate work on the data centre refresh, getting under control. FJ running this well, risk review & declaration every month. RH will kick again but doing as much as we can. • Credence - Support and maintenance has moved to Accenture but on old hardware. Single platform if goes, goes completely. Driving pressure into Accenture to get more capability. Accenture picked up difficult pass and the additional costs at the moment. Starting to see light at the end of the tunnel. RH having difficult conversations, flag up any issues, assumptions and risks to GE. KPMG engaged to undertake a GAP analysis to produce an IT control framework. • What are the risks of the 'hop' to thin client not working? RH - need to prove thin client works and the 'hop' is a safe way to test we could revert to HNGA if the 'hop' fails. All nervous as we are dealing with legacy systems. More comfortable that we are not changing Back & Front office at the same time, but still using Horizon. SSKs have proven that thin client can work, it is the journey to implement that is vital. • Fujitsu - negotiations will be tough as POL do not hold the power. FJ see the contract as a cash cow, so need to persuade them that working with POL to migrate to cloud technology is to their benefit against a 'too good' contract. They may say make us whole but we cannot give additional work because of the procurement risk. Only choice if they will not work with us is to start building an alternative. Come back to GE with an update on the FJ negotiations plan”
“However, we need to continue to manage firmly any over-expectations of the frontline that Horizon Online will deliver improved functionality - they may see this as a missed opportunity so will not cure all the issues and problems that users have with Horizon although where practical, and at no extra cost, we have smoothed away a number of "rough edges". (emphasis added) There was no significant upgrade or improvement in functionality intended as part of HNG-X. 943. Another type of document that is illuminating in terms of the Horizon Issues are internal Fujitsu briefing documents, and to a lesser extent also some internal Post Office emails, that show the same approach to, or view of, Horizon within those organisations as are drawn by my conclusions. One example relates to the so-called Ping fix. After this was brought in, the number and value of TCs fell dramatically so far as Camelot/the lottery was concerned. This appears at [427] to [428] of the judgment on the Common Issues, Judgment (No.3). In 2007 there had been the following value of TCs in respect of the lottery: approximately (to 1 decimal point, which represents£100 thousand )£22.8 million ; in 2008 there were£12.5 million ; in 2009 there were£12.0 million ; in 2010 this became£11.3 million ; and in 2011£4.5 million . In 2012, the year the Ping fix was introduced, these fell below£1 million for that year. 944. The Introduction to a Post Office document of 2011 entitled “Transaction Acknowledgements – End to End Incident Management and Data Flow” explains the following purpose to the Ping fix: “Introduction The PING Project was part of the Back Office Efficiency Programme within Post Office Ltd. The objective of this project was to automate a process that converts Client transactional data files (e.g. Paystation), into a data feed to the branch, i.e. a Transaction Acknowledgement file. The branch will then process the data electronically on Horizon Online and thus avoid non compliance issues and the resulting cost to the business to process queries as a result. Please note - The term PING is an internal Post Office Ltd term and branches and helplines will know this new functionality as 'Transaction Acknowledgements'.”
" Failure to be open and honest when issues arise eg roll out of Horizon, HNGx migration issues/issues affecting few branches not seemingly publicised."
“Subject: The Dalmellington Error in Horizon / problemswithpol Dear both, This needs looking into please. https://problemswithpol.wordpress.com/2015/11/10/the-error-in- horizon/?iframe=true&theme_preview=true [This is a reference to an internet post about a conviction of an SPM] Can I have a report that takes the points in order and explains them. Tim McCormack is campaigning against PO and Horizon. I had another note from him this am which Tom will forward, so you are both in the loop. We must take him seriously and professionally. This particular blog is independent of Sparrow but clearly related in that it appears to present similar challenges that were raised in the course of the scheme. I'm most concerned that we/our suppliers appear to be very lax at handling£24k . And want to know we've rectified all the issues raised, if they happened as Tim explains. Thanks. Paula” 949. Mr Houghton of the Post Office sent it on to someone else on the same day, copied to Mrs Van Den Bogerd. This stated: “I need an urgent review and mini <taskforce> on this one. It probably needs to link up heavily with Angela’s work as FSC are mentioned extensively - Angela cfi. I don't know how we respond to this but can we section a few inside people to get all over it and give me/Al/ Paula evidence and understanding.” 950. However, the same recipients also received the following message from Mr Houghton later on the very same day: “Can you stand down on this please? [A redacted section then follows] Any specific actions and I will revert. My apologies.” 951. The email chain is heavily redacted and therefore the reasoning behind this volte face within the Post Office is not shown, nor was it explained by any Post Office witness. 952. Thus the entirely understandable initial reaction of the Chief Executive of the Post Office, that a point by point investigation was required in respect of specific criticisms of Horizon in respect of one particular SPM who blamed Horizon for shortfalls and discrepancies, appears to have been very swiftly shut down and not pursued due to a decision at what must have been very high levels within the Post Office. I can think of no justifiable reason why the Post Office, institutionally, would not want to address the Chief Executive’s points and investigate as she initially intended, and find out for itself the true situation of what had occurred. 953. Both the recipients of the Chief Executive’s email of1 July 2016 above are specifically named in the Technology Strategy Update Decision Paper of30 January 2017 in which the following is stated specifically. Mr Houghton is the Author of the paper, and Mr Al Cameron is “the Sponsor”
“This document forms an update to the IT Strategy approved in July 2016 by the PO Board. In July we outlined that IT was not fit for purpose , expensive and difficult to change.”
“….even the documents that your Lordship saw this morning with the Dalmellington bug, your Lordship will see the rigour that's applied. There's concern that postmasters aren't given advice that might be incorrect. The rigour associated with that process and the determination of Fujitsu and the other people involved, the other stakeholders, to get to the bottom of what happened is quite striking in my submission.” 956. Dalmellington was investigated in 2015, and the outcome of that investigation is available to this court. The outcome of that investigation is far from favourable to the Post Office’s position on bugs, errors and defects, and the robustness of Horizon. The oral submission that the Dalmellington bug was investigated rigorously (which it was) does not easily sit with the documents showing that one year later the Post Office specifically decided not to investigate other incidents at all, and to “stand down” those tasked with investigating it. This is notwithstanding that the Chief Executive’s initial (and in my judgment entirely proper, if not the only proper) reaction was that those allegations needed to be looked at “seriously and professionally”, that an investigation was needed, and that an urgent review and mini task force was required. Nor does it sit easily with the way that other bugs that were present in both Legacy Horizon and Horizon Online were investigated by Fujitsu. 957. Further, a conclusion in terms that the IT is “not fit for purpose” is not something that would have been reached lightly in an IT strategy document, or approved lightly by the Post Office Board. The Post Office Board is a serious level within the organisation. The Board are not likely to be involved in, nor to have brought to their attention, matters that are anything other than serious, considered and fully researched. This conclusion in this internal contemporaneous document has not driven my conclusions on the evidence in the slightest. Rather, it is the other way around. I have reached my conclusions on the evidence independent of contemporaneous references such as this one, but then tested my evidential conclusions against these to see if there is a contradiction. However, there is none, and they are entirely consistent. 958. There are only very isolated Post Office documents that are inconsistent with my conclusions. I shall deal with only one. On17 August 2015 the BBC Panorama programme broadcast a programme which contained allegations about Horizon. The contemporaneous response to that by the Post Office in its statement to SPMs started with the following passages, which I shall reproduce. The bold emphasis was present in the original: “The Post Office wholly rejects extremely serious allegations repeated in BBC’s Panorama programme of17 August 2015 . The allegations are based on partial, selective and misleading information. · The Post Office does not prosecute people for making innocent mistakes and never has · There is no evidence that faults with the computer system caused money to go missing at these Post Office branches · There is evidence that user actions, including dishonest conduct, were responsible for missing money ” 959. The first of those bullets point is for another day, and is not for this court in any event, for the reasons explained at [60] to [66] above. This judgment deals with the detailed background and technical evidence behind the second bullet point. The confidence expressed by the Post Office in 2015, in that second bullet point, in the computer system and its accuracy is not consistent with the evidence I have accepted, the expert evidence on the number of bugs in the Horizon system (both Legacy Horizon and HNG-X) and my findings. This Post Office document was however drafted for public consumption, and this must explain the inconsistency. To the extent that the second bullet point conflicts with my findings in this judgment, that public statement in 2015 by the Post Office is factually incorrect. This is not to make any findings on any individual claim, or on the cause of money said to be missing in the accounts of any particular branch Post Office or individual claimant SPM. 960. In my judgment, a number of both the Post Office’s own internal documents, and Fujitsu ones too, namely those that were not drafted for public consumption, plainly support my conclusions on the evidence. Further, certain matters that have emerged in the Horizon Issues trial – such as discussions within Fujitsu itself as to whether the Post Office should be told certain detrimental information about the Horizon system, and the Post Office’s own decision at the highest level not to investigate certain matters as recently as 2016 – are of great concern. The Post Office has gone to great lengths over the years, and spent a great deal of time and a huge amount of public money, in defending the performance of Horizon. It is also the case that the Post Office must have been reliant on Fujitsu to a certain degree in terms of being provided with accurate information of a technical nature. That is not only obvious from the evidence, but has also been agreed by the experts in the 3 rd Joint Statement. That accuracy from Fujitsu has not always been available, as demonstrated by this judgment. 961. However, regardless of that – and this judgment does not deal with who, if anyone, at the Post Office knew precisely what about Horizon, and when, as these are not part of the Horizon Issues – in my judgment a number of the specific internal contemporaneous documents, rather than being contrary to my conclusions on the evidence, are entirely consistent with those conclusions. I do not know if the Post Office Board, who were told in July 2016 that its own IT was not fit for purpose, will be particularly surprised at the findings on the expert evidence in the Bug Table, and the number of admitted bugs. It does not much matter, for the purposes of resolving the Horizon Issues, whether they are surprised or not. 962. The experts’ agreements in particular have been of great assistance, but everything has been considered. This judgment, together with the Technical Appendix, is of substantial length, and to recite everything that has been deployed by both sides, in terms of evidence, submission and reference to all contemporaneous documents, is neither necessary nor desirable. 963. It must also be remembered that Horizon as it is today, or at least in the last couple of years since it became HNG-A, is a very different system to earlier times. Legacy Horizon ceased to be used in 2010. Modern Horizon Online as it is today is not the same as the online system that was introduced in 2010; it is different and more robust, on the agreed expert evidence, to the system as it was even when the litigation started with the issue of the first claim form on11 April 2016 . It became HNG-A in 2017, rather than HNG-X, and runs on a completely different Windows platform. The dates of the bugs in the Bug Table are (between them collectively) from 2000 to 2018, although there are far fewer in recent years. Having said that, some – such as Bug 8, Recovery Issues, and Bug 28, Drop and Go, have been experienced up to 2018 and 2017 respectively. The latter is a reference data bug and hence easily fixed. The Technical Appendix at [421] provides a summary of the different bugs from Legacy Horizon and Horizon Online. It can be seen that there were more in Legacy Horizon than HNG-X in any event, and far fewer in HNG-A. In any case, the cut-off date for claims to be issued in the group litigation is24 November 2017 , as extended by paragraph 15 of the 1 st Case Management Order sealed on27 October 2017 . 964. The Post Office has been very concerned as to the outcome of this litigation on its wider business. It described the group litigation as long ago as its Opening Submissions in November 2018 for the Common Issues trial as representing “an existential threat” to that business. I do not consider that these answers to the Horizon Issues represent a significant threat to the Post Office’s entire business. Findings in this judgment as to the performance and robustness of Legacy Horizon from 2000 to 2010, and then of Horizon Online (in both its forms, HNG-X and HNG-A) from 2010 to 2018 are not findings on the Horizon system as it exists at the date of this judgment. These findings cannot be routinely applied to the way that HNG-A operates as at December 2019. It is agreed by the experts that the Horizon System in its HNG-A form is now relatively robust. This judgment is a historical analysis of the Horizon System as it relates to the period in question in the group litigation, not a judgment upon Horizon HNG-A as it is today. M. Answers to the Horizon Issues 965. This section is intended as a very brief summary of what has gone before, both in this judgment and also the Technical Appendix. It does not replace the previous 964 paragraphs or that appendix. The detailed passages themselves should be consulted for a more detailed consideration of the issues. 966. I therefore turn to the answers to the Horizon Issues themselves. For convenience, I shall reproduce the specific issue above each answer, in order to avoid the reader having constantly to turn back some hundreds of pages to remind themselves. I shall also retain the headings and sub-headings. The wording of the Horizon Issues was agreed by the parties and approved by the court. BUGS, ERRORS AND DEFECTS IN HORIZON Accuracy and integrity of data 967. Issue (1): To what extent was it possible or likely for bugs, errors or defects of the nature alleged at §§23 and 24 of the GPOC and referred to in §§49 to 56 of the Generic Defence to have the potential (a) to cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, or (b) undermine the reliability of Horizon accurately to process and to record transactions as alleged at §24.1 GPOC? 968. Answer: It was possible for bugs, errors or defects of the nature alleged by the claimants to have the potential both (a) to cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, and also (b) to undermine the reliability of Horizon accurately to process and to record transactions as alleged by the claimants. 969. Further, all the evidence in the Horizon Issues trial shows not only was there the potential for this to occur, but it actually has happened, and on numerous occasions. This applies both to Legacy Horizon and also Horizon Online. It has happened under both the HNG-X and HNG-A iterations of the Online system, but far less frequently under the latter than the former. Indeed, there are only isolated instances of it happening in respect of HNG-A, which the experts agree is a better system than either of the other two iterations of Horizon. 970. I accept the claimants’ submissions that, in terms of likelihood, there was a significant and material risk on occasion of branch accounts being affected in the way alleged by the claimants by bugs, errors and defects. This is amply demonstrated by Dr Worden’s evidence. He accepted that there was strong evidence of at least 12 bugs causing a lasting discrepancy in branch accounts. This conclusion was reached even within the context of his unjustified creation of both “transient” and lasting impact, which limited his consideration of this, and is an approach which I have rejected. This number of bugs was very much smaller than the number contended for by Mr Coyne, and the number which I have found in Part E of the Technical Appendix, but even on that smaller number, was itself significant. Dr Worden’s reliance upon TCs as a countermeasure led to a failure by him to follow the agreed wording of the Horizon Issues, as TCs are not part of the Horizon system. Given TCs are part of the way that the Post Office corrects the branch accounts (and branch accounting statements) produced by Horizon, the only way that a fully precise answer could be given for the number of times such impacts have occurred would be if the figures were available for the years of operation of Horizon, demonstrating how many TCs each year had to be issued in respect of the effects on branch accounts of such discrepancies. Figures would also be required of the number of TCs that were upheld when disputed by SPMs. 971. However, those figures are simply not available for the majority of the life of both Legacy Horizon and Horizon Online. Nor is there any data centrally recorded by Fujitsu of the number of bugs, errors and defects which were discovered by dissatisfied SPMs reporting incidents to Fujitsu, with percentage outcomes of what the investigations by the SSC led to, in terms of result. Such data as there is, namely Mr Parker’s summation of closure codes of PEAKs, is wholly unreliable and I have rejected it. Nor are the closure codes that were used by Fujitsu accurate in any event. The evidence of fact was that full and accurate records of this nature are simply not kept by Fujitsu. Had they been, both experts could (and probably would) have used such data as a starting point for their investigations. The sheer scale of the number of TCs issued by the Post Office each year – which is over one hundred thousand for many of the years that are the subject of the group litigation – supports my conclusion that there was a significant and material risk of inaccuracy in branch accounts as a result of bugs, errors and defects in the Horizon System (both Legacy Horizon and HNG-X). 972. Issue (2): Did the Horizon IT system itself alert Subpostmasters of such bugs, errors or defects as described in (1) above and if so how? 973. Answer: Although the experts were agreed that the extent to which any IT system can automatically alert its users to bugs within the system itself is necessarily limited, and although Horizon has automated checks which would detect certain bugs, they were also agreed that there are types of bugs which would not be detected by such checks. Indeed, the evidence showed that some bugs lay undiscovered in the Horizon system for years. This issue is very easy, therefore, to answer. The correct answer is very short. The answer to Issue 12 is “No, the Horizon system did not alert SPMs”