15 replies to this topic

[Helmut Schütz]

Dear all,

EMA's "Questions & Answers: Positions on specific questions addressed to the Pharmacokinetics Working Party" EMA/618604/2000 Rev. 3, dated 26 January 2011 was published on 14 March 2011.

The interesting part is Section 11. Clarification on the recommended statistical method for the analysis of a bioequivalence study (pages 21-32).

See also:
3.3. Alternative computer programs
SAS (version 9.1, SAS Institute Inc, NC) was used in the previous computations. Results obtained by alternative, validated statistical programs are also acceptable […].

Unless we have found out a way to verifiy EMA's results, we cannot use Phoenix for regulatory applications in the EU!

For your convenience I've attached a project containing EMA's example data sets (full replicate and partial replicate).

Open questions:
Coding for Methods A and B
Calculation of within subject CV needed for scaling

[file name=EMA_Replicate_Data_Sets.phxproj size=46486]http://www.pharsight.com/extranet/media/kunena/attachments/legacy/files/EMA_Replicate_Data_Sets.phxproj[/file]

[Helmut Schütz]

Dear all,

 

I could reproduce EMA's results when setting up LME from scratch. Surprisingly there are some minor differences in results if I misuse (i.e. delete components) from PHX's default BE model(s).

 

Attached the project and a comparison in Excel. [file name=EMA_Replicate.phxproj size=2506892]http://www.pharsight.com/extranet/media/kunena/attachments/legacy/files/EMA_Replicate.phxproj[/file] [file name=Validation_Replicate_Design_EMA.xls size=15872]http://www.pharsight.com/extranet/media/kunena/attachments/legacy/files/Validation_Replicate_Design_EMA.xls[/file]

Attached Files

•  Validation_Replicate_Design_EMA.xls   15.5KB   1086 downloads

[Helmut Schütz]

Dear all,

I was able to track down the discrepancy between a model set up in LME from scratch and a modified BE model.
Example: EMA's data set I (full replicate, imbalanced and missing observations), all data in full precision.
BE gives for LSM Differences:
R-T (idiotic, because reference is mapped to R): -0.145473721369306
Lower_CI: -0.222301703008674
Upper_CI: -0.068645739729938
and in Average Bioequivalence: (antilogged and changed signs!)
Difference: 0.145473721369306
Ratio_Ref: 115.658734065124 (rounded 115.66)
CI_90_Lower: 107.103869746113 (rounded 107.10)
CI_90_Upper: 124.89691359665 (rounded 124.90)
But if I change signs, and antilog results of LSM Differences I get:
Ratio_Ref: 115.658734065125 (rounded 115.66)
CI_90_Lower: 107.10567090924 (rounded 107.11)
CI_90_Upper: 124.894813243667 (rounded 124.89)

Only the latter agrees with results from LME and results reported by EMA (obtained in SAS 9.21). Since according to the guideline the CI has to be rounded to two decimal places, this discrepancy is not nitpicking - it might kill a study! I will file an incident at the support site. In the meantime, I would suggest to calculate the CI from the LSM Differences rather than relying on the flawed internal back-transformation in the Average Bioequivalence output.

I checked a couple of 'real world' studies as well (conventional 2×2 cross-over and 3-period, 6-sequence Williams' designs). The CI given in ABE output and calculated from LSM was different in all (!) cases. There was no clear trend in the size of the CI; the CI was wider in ~50% of cases and narrower in ~50%. However, the largest difference in widths I have seen was 0.82%!

[Emily Colby]

Dear Helmut,

 

Thank you for doing this work. I have been assigned to investigate this further.

 

We know about your request to have the LSM Differences calculated based on the appropriate reference treatment. This part of the BE program (the part that calculates the other worksheets such as LSM differences), called LinMix, which has been virtually untouched for years, works pretty much the same way as SAS PROC MIXED and thus does not have the reference treatment as an input.

 

The Average Bioequivalence worksheet is calculated by a different program that does post-processing of the LinMix output and assigns the reference treatment appropriately. Since the pertinent output is the Average Bioequivlance worksheet, we have not seen the request to change the reference value in LinMix to be a priority. However, I can understand your frustration if you have been going by the LSM differences worksheet for the appropriate level of precision.

 

Let me ask you this: If we were to increase the precision of the estimates (including CI's) in the Average Bioequivalence worksheet, would this address the problem?

 

Best regards,

Emily

[Helmut Schütz]

Hi Emily!

The Average Bioequivalence worksheet is calculated by a different program that does post-processing of the LinMix output and assigns the reference treatment appropriately.

Right. If I look at the LSM Differences in ABE I get excatly the same values as if I set up LME from scratch.
But: Obviously the back-transformation (giving the CI in ABE Results) is flawed.

Since the pertinent output is the Average Bioequivlance worksheet, we have not seen the request to change the reference value in LinMix to be a priority. However, I can understand your frustration if you have been going by the LSM differences worksheet for the appropriate level of precision.
Let me ask you this: If we were to increase the precision of the estimates (including CI's) in the Average Bioequivalence worksheet, would this address the problem?

It's not an issue of precision; values are simply wrong. With EMA's dataset the width of the CI is wrong by only 0.02% maximum, but I have seen other datasets, where the difference was 0.82%. I didn't see a pattern: wider/narrower in ~50% of cases. This might either kill a study (if the CI is too wide; BE not demonstrated) or may lead to registration of a formulation, which is not BE (narrower CI, actual patients' risk >5%).

This is a major problem!

[Emily Colby]

Dear Helmut,

 

I suspect there may be rounding going on in more than one place, which could lead to the level of discrepancy you're talking about in some cases. I will pass this on to our developers.

 

Thank you,

Emily

[Emily Colby]

This has been entered as a bug (QC 10856).

[Helmut Schütz]

Dear Emily,

THX! I will post a white paper concerning the validation in the next couple of days.
In the meantime I would suggest everybody not to use standard output of ABE any more, but use following workaround:

Set up a series of Custom Column Transformations (assuming log-transformed data)
Map ABE results - LSM Differences worksheet
New Column Name AntlogPE
Formula round(100*exp(-Estimate),2)
New
New Column Name AntilogLower_CI
Formula round(100*exp(-Upper_CI),2)
New
New Column Name AntilogUpper_CI
Formula round(100*exp(-Lower_CI),2)

[Emily Colby]

Dear Helmut,

 

I just had a thought. Our confidence intervals are on the percent scale, so that it reports, for example, 80.00 and 125.00. If you're looking for two decimal places on the percent scale, this could mean four or five significant figures. Are you sure that the EMA is expecting four or five significant figures? A study with a sample size of 20 is probably not likely to guarantee that much precision anyway. I suspect they may have been referring to rounding to two decimal places if you are on the original scale, for example, 0.80 and 1.25.

 

What are your thoughts?

 

Emily

[Helmut Schütz]

Dear Emily!

 

Nice try. ;-)

EMA - like the FDA - specificaly asks for the CI rounded to two decimal places in percent to be included in the acceptance range of 80.00-125.00%.

See FDA:

http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/ucm070124.pdf (page 26)

and EMA:

http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2010/01/WC500070039.pdf (page 15).

 

[Edit]

In the meantime I compared PHX's results of example data set 'Data 2x2.csv' with routines according to Chow&Liu implemented in R and HTBasic. The CI of both my routines and PHX back-transformed CI from LMEwas identical to 7 significant digits; the ABE CI differed in the 5th significant figure.

[Helmut Schütz]

Dear all,

 

I prepared a white paper suggesting a workaround.

Scaling according to FDA's current requirements is beyond my skills. :-( Any ideas appreciated.

 

http://bebac.at/downloads/Replicate%20Designs%20for%20ABE%20according%20to%20EMA%20with%20Phoenix%20v2.3.pdf (1.08MB PDF)

[Helmut Schütz]

Hi Emily!

Any news? In the meantime I played around with some datasets. In an extreme one (2×2 cross-over, 4 (!!) subjects, imbalanced 3:1, very low CVintra of 7.99%) the 90%CIs from Bioequivalence:
78.22–114.31% and from the workaround based on LSM:
73.27–122.03%…

I can’t imagine that the reason is a double rounding in post-processing of LSM to BE alone.

[Ana Henry]

This is a follow up on Track ID QC 10853 mentioned in this forum post. This issue has been corrected in Phoenix 1.3 (Phoenix WinNonlin 6.3) to be released Q1/2012.

 

In Phoenix 1.2.1 and prior, the LowerCI and UpperCI values in the LSM Differences output worksheet (or the exponential of these values if ln-transform was used in Bioequivalence) did not exactly match the values for the confidence intervals shown in the Average Bioequivalence output worksheet. The differences would occur at about the fifth significant digit. Computing these confidence intervals requires estimating a Student's t value, and the reason for the difference between the worksheets was that the t-inverse estimation routine in LinMix (for LSM Differences) was different than the t-inverse estimation routine used in the Bioequivalence code. Bioequivalence now uses the slightly more accurate LinMix code, so accuracy is improved at about the fifth significant digit and the values will match between the two worksheets.

 

Ana Henry

Pharsight Product Manager

[belen sadaba]

Any progress on this issue?

I would like to know how to calculate the intra-individual variability in a semireplicated bioequivalence study (three way), giving two times the reference medication in three periods with WinNonlin. I understand that I should be able to do it with the data replicated from the reference formulation, but I can not find how.

 

Thank you

Belén

[Helmut Schütz]

Dear Belén!

 

I guess the CI-problem will not be corrected in WinNonlin. It’s important which regulatory model you are aiming at. If you are going with FDA’s see the progesterone guidance and adapt the given SAS-code in LME. If you are going with EMA’s see my whitepaper (link in a previous post above).

I would suggest to upgrade to Phoenix/WinNonlin; it’s much easier there.

[Helmut Schütz]

Since the bug was corrected in v1.3 the workaround is not needed any more. Due to the new data wizard the workflow is much easier to set up (THX to Simon for support!).

See the latest version of the white paper:

http://bebac.at/downloads/Replicate%20Designs%20for%20ABE%20according%20to%20EMA%20with%20Phoenix%20v3.pdf