CharDistributionAnalysis Class Reference

#include <CharDistribution.h>

Inheritance diagram for CharDistributionAnalysis:

[legend]

Public Member Functions
	CharDistributionAnalysis ()
virtual	~CharDistributionAnalysis ()
void	HandleData (const char *aBuf, PRUint32 aLen)
void	HandleOneChar (const char *aStr, PRUint32 aCharLen)
float	GetConfidence ()
void	Reset (void)
void	SetOpion ()
PRBool	GotEnoughData ()
Protected Member Functions
virtual PRInt32	GetOrder (const char *str)
Protected Attributes
PRBool	mDone
PRUint32	mFreqChars
PRUint32	mTotalChars
const PRInt16 *	mCharToFreqOrder
PRUint32	mTableSize
float	mTypicalDistributionRatio

---

Detailed Description

Definition at line 45 of file CharDistribution.h.

---

Constructor & Destructor Documentation

CharDistributionAnalysis::CharDistributionAnalysis

(

)

[inline]

Definition at line 48 of file CharDistribution.h.

References Reset().

{Reset();}

Here is the call graph for this function:

virtual CharDistributionAnalysis::~CharDistributionAnalysis

(

)

[inline, virtual]

Definition at line 49 of file CharDistribution.h.

{}

---

Member Function Documentation

float CharDistributionAnalysis::GetConfidence

(

)

Definition at line 52 of file CharDistribution.cpp.

References mFreqChars, MINIMUM_DATA_THRESHOLD, mTotalChars, mTypicalDistributionRatio, SURE_NO, and SURE_YES.

Referenced by nsSJISProber::GetConfidence(), nsGB18030Prober::GetConfidence(), nsEUCTWProber::GetConfidence(), nsEUCKRProber::GetConfidence(), nsEUCJPProber::GetConfidence(), and nsBig5Prober::GetConfidence().

{ 
  //if we didn't receive any character in our consideration range, or the
  // number of frequent characters is below the minimum threshold, return
  // negative answer
  if (mTotalChars <= 0 || mFreqChars <= MINIMUM_DATA_THRESHOLD)
    return SURE_NO;

  if (mTotalChars != mFreqChars) {
    float r = mFreqChars / ((mTotalChars - mFreqChars) * mTypicalDistributionRatio);

    if (r < SURE_YES)
      return r;
  }
  //normalize confidence, (we don't want to be 100% sure)
  return SURE_YES;
}

Here is the caller graph for this function:

virtual PRInt32 CharDistributionAnalysis::GetOrder

(

const char *

str

)

[inline, protected, virtual]

Reimplemented in EUCTWDistributionAnalysis, EUCKRDistributionAnalysis, GB2312DistributionAnalysis, Big5DistributionAnalysis, SJISDistributionAnalysis, and EUCJPDistributionAnalysis.

Definition at line 97 of file CharDistribution.h.

Referenced by HandleOneChar().

{return -1;};

Here is the caller graph for this function:

PRBool CharDistributionAnalysis::GotEnoughData

(

)

[inline]

Definition at line 91 of file CharDistribution.h.

References ENOUGH_DATA_THRESHOLD, and mTotalChars.

Referenced by nsGB18030Prober::HandleData(), nsEUCTWProber::HandleData(), nsEUCKRProber::HandleData(), and nsBig5Prober::HandleData().

{return mTotalChars > ENOUGH_DATA_THRESHOLD;};

Here is the caller graph for this function:

void CharDistributionAnalysis::HandleData	(	const char *	aBuf,
		PRUint32	aLen
	)			[inline]

Definition at line 52 of file CharDistribution.h.

{};

void CharDistributionAnalysis::HandleOneChar	(	const char *	aStr,
		PRUint32	aCharLen
	)			[inline]

Definition at line 55 of file CharDistribution.h.

References GetOrder(), mCharToFreqOrder, mFreqChars, mTableSize, and mTotalChars.

Referenced by nsSJISProber::HandleData(), nsGB18030Prober::HandleData(), nsEUCTWProber::HandleData(), nsEUCKRProber::HandleData(), nsEUCJPProber::HandleData(), and nsBig5Prober::HandleData().

  {
    PRInt32 order;

    //we only care about 2-bytes character in our distribution analysis
    order = (aCharLen == 2) ? GetOrder(aStr) : -1;

    if (order >= 0)
    {
      mTotalChars++;
      //order is valid
      if ((PRUint32)order < mTableSize)
      {
        if (512 > mCharToFreqOrder[order])
          mFreqChars++;
      }
    }
  };

Here is the call graph for this function:

Here is the caller graph for this function:

void CharDistributionAnalysis::Reset

(

void

)

[inline]

Definition at line 78 of file CharDistribution.h.

References mDone, mFreqChars, mTotalChars, and PR_FALSE.

Referenced by CharDistributionAnalysis(), nsSJISProber::Reset(), nsGB18030Prober::Reset(), nsEUCTWProber::Reset(), nsEUCKRProber::Reset(), nsEUCJPProber::Reset(), and nsBig5Prober::Reset().

  {
    mDone = PR_FALSE;
    mTotalChars = 0;
    mFreqChars = 0;
  };

Here is the caller graph for this function:

void CharDistributionAnalysis::SetOpion

(

)

[inline]

Definition at line 87 of file CharDistribution.h.

{};

---

Field Documentation

const PRInt16* CharDistributionAnalysis::mCharToFreqOrder [protected]

Definition at line 109 of file CharDistribution.h.

Referenced by Big5DistributionAnalysis::Big5DistributionAnalysis(), EUCJPDistributionAnalysis::EUCJPDistributionAnalysis(), EUCKRDistributionAnalysis::EUCKRDistributionAnalysis(), EUCTWDistributionAnalysis::EUCTWDistributionAnalysis(), GB2312DistributionAnalysis::GB2312DistributionAnalysis(), HandleOneChar(), and SJISDistributionAnalysis::SJISDistributionAnalysis().

PRBool CharDistributionAnalysis::mDone [protected]

Definition at line 97 of file CharDistribution.h.

Referenced by Reset().

PRUint32 CharDistributionAnalysis::mFreqChars [protected]

Definition at line 103 of file CharDistribution.h.

Referenced by GetConfidence(), HandleOneChar(), and Reset().

PRUint32 CharDistributionAnalysis::mTableSize [protected]

Definition at line 112 of file CharDistribution.h.

Referenced by Big5DistributionAnalysis::Big5DistributionAnalysis(), EUCJPDistributionAnalysis::EUCJPDistributionAnalysis(), EUCKRDistributionAnalysis::EUCKRDistributionAnalysis(), EUCTWDistributionAnalysis::EUCTWDistributionAnalysis(), GB2312DistributionAnalysis::GB2312DistributionAnalysis(), HandleOneChar(), and SJISDistributionAnalysis::SJISDistributionAnalysis().

PRUint32 CharDistributionAnalysis::mTotalChars [protected]

Definition at line 106 of file CharDistribution.h.

Referenced by GetConfidence(), GotEnoughData(), HandleOneChar(), and Reset().

float CharDistributionAnalysis::mTypicalDistributionRatio [protected]

Definition at line 116 of file CharDistribution.h.

Referenced by Big5DistributionAnalysis::Big5DistributionAnalysis(), EUCJPDistributionAnalysis::EUCJPDistributionAnalysis(), EUCKRDistributionAnalysis::EUCKRDistributionAnalysis(), EUCTWDistributionAnalysis::EUCTWDistributionAnalysis(), GB2312DistributionAnalysis::GB2312DistributionAnalysis(), GetConfidence(), and SJISDistributionAnalysis::SJISDistributionAnalysis().

---

The documentation for this class was generated from the following files:

• CharDistribution.h
• CharDistribution.cpp