Neural Network Classification and Prior Class Probabilities

A commonly encountered problem in MLP (multi-layer perceptron) classification problems is related to the prior probabilities of the individual classes if the number of training examples that correspond to each class varies significantly between the classes, then it may be harder for the network to learn the rarer classes in some cases. Such practical experience does not match theoretical results which show that MLPs approximate Bayesian a posteriori probabilities (independent of the prior class probabilities). Our investigation of the problem shows that the difference between the theoretical and practical results lies with the assumptions made in the theory (accurate estimation of Bayesian a posteriori probabilities requires the network to be large enough, training to converge to a global minimum, infinite training data, and the a priori class probabilities of the test set to be correctly represented in the training set). Specifically, the problem can often be traced to the fact that efficient MLP training mechanisms lead to sub-optimal solutions for most practical problems. In this chapter, we demonstrate the problem, discuss possible methods for alleviating it, and introduce new heuristics which are shown to perform well on a sample ECG classification problem. The heuristics may also be used as a simple means of adjusting for unequal misclassification costs. © Springer-Verlag Berlin Heidelberg 2012. Keywords prior, class, probabilities, classification, network, neural Disciplines Engineering | Science and Technology Studies Publication Details Lawrence, S., Burns, I., Back, A., Tsoi, A. Chung. & Giles, C. Lee. (1998). Neural network classification and prior class probabilities. Lecture Notes in Computer Science, 7700 LECTURE NO 299-314. This journal article is available at Research Online: http://ro.uow.edu.au/eispapers/271 !" #$ % !& ' # ( ) #* !+, !#$!#* . / 01 2345,+6 #* 786 * #*/ *#$8 !& !# 9: 9: !",# 9;<. .86+ =>?# @6 A( #*', !#*'BC?@BDEFGH$ & / 'IJ F 9;KLF MNO +,P P #* Q&2/,'BKLF 01&2 +,&2/,&6(G865 /,R2#* 1ST#* P &2R ( 5 5GF U2V VWX Y ZW[ ( ZWV2V \ F ] _̂`abc4defg] _̂ h.ijklc4mnojf6dNp pqr stu dvhqr,kwxydexbz{|}c~r6kwcojf,dNpEp |€c4k‚bde‚ƒr,f,r hqr,_„p …†‡ ˆ ‰%ˆNŠ‹ŒG Ž2ˆ,76ˆ ZG‘1’ Œ b“B”1Ž 7• U%‘1– Q—TŽ2ˆ,˜“™Œ%?š Y›‘Q–`œžIœ ”Q1Ÿ ¡ • ¢ £ [ ‘ ž¤¥ Š$ˆ ˆN¦v£;§;ˆ • Z? ̈ © a:«?¬4­W®?̄ °?® ± 2 3 a ®? ́ μ,¶I­W®?́*® «,­W° ·7 ̧ ̄,12̧ ̄†®?° ̧ °2o »1⁄4±73:«,̄7 ̧ 1⁄2†3⁄4%­W ̄†́ ¶Bo «% ̧ °?o »1⁄4̧ «,3⁄47± 1⁄2¿« ° À ¶BÁ?o›o̧ ­W3 ÀG®? ̄Q̧ 2 o̧ 1AÂ7± à · Ä · 3⁄4† ̄ 2 Åq ́*o 3:¶B3⁄4†o ¬ ̧ ® Æ 3⁄4Q ̧ « 3⁄4 Ç~骃 ÄÊËI®? ́*®?«,­W° ·1⁄4ÌA ̄́ ÍW3 ÍW3⁄4%ÍW® Î Î ±†ÏÐÌ ̄¿Æ%®?ÂT®? ̄¿Æ%®?̄†°2®Iу«?қ± ÓL­W3Ô ̄ °?®2ÍWo ̄7± șÕNÖ × Ø Ï Ö Ù Ú ÂT®2 ̄ à °2°?®? ́* ́ÛÓ$Í ÒwÜQÍ ÆQ±†Ü7®2ÝG® ?a7Þ ±Tß à à a 1⁄2¿«, ̄ ҁ ›Í ±T ›Í ̧†ÜQ®?o ̄†«,­ Æ% ́?± È ›Ñ Þ Ö GØ ± à 3⁄4 ́ Í*­ « a 3Ô« ­ « 3 ̄ Ì ̄ o,­W»e«,ÍW3 o ̄Ó$­Wo›°2®?́* ́*3 ̄†2w I­Wo 3⁄4 Â7±†Åq·†® ÌA ̄ ́ ÍW3 ÍW3⁄4 ÍW®™o, $Ӌ·›Ò% ́*3 ° « a.« ̄†Æ1⁄4Ä ·†®2»N3Ô°?« a ËÛ®?́*® «6­W° · =ËIÌ É‹È™ ± Õ « ¿«, ̄ „ « °23⁄4†a ÍAÒlo ÌA ̄ ;o ­W»e«6ÍW3Ô°2 ́?±† I ̄†3 ÝG®2­W ́*3 ÍAÒlo, LÑbo a a o ̄ 2 o ̄†2 ± È o,­*ÍW· ¿®2a:Æ% ́ à ÝG® ±%Ñ o a a o ̄ 2 o ̄ 2 ± È Ñ Þ Ø Þ Þ ± à 3⁄4† ́ Í*­ « a 3Ô« l L ? 2 , † à °?o »N»No ̄†a ÒЮ?̄ °?o 3⁄4 ̄›ÍW®2­W® ÆN ­Wo 1⁄2†a ®?» 3 ̄ 1⁄4Ü.Ó =»v3⁄4 a ÍW3 !aÔ«?қ®2­EÂT®2­* °?®? Í*­Wo ̄ °?aÔ« ́* ́*3 ¿° «6ÍW3 o ̄ÊÂ%­Wo 1⁄2†a ®?»Ńv3Ô ́¤­W®?aÔ«,ÍW® ÆtÍWo ÍW· ®wÂ%­W3Ôo,­e ­Wo 1⁄2¿« 1⁄2 3 aÔ3 ÍW3 ®? ́No, ÍW·†®Ð3 ̄¿Æ%3 Ý%3ÔÆ%3⁄4¿« a‹°?aÔ« ́*́*®2 ́ à̀ 3 4ÍW·†®v ̄%3⁄4 »v1⁄2T®2­™o 4Í*­ « 3 ̄ 3Ô ̄ 2 ® « »N aÔ®2 ́™ÍW·¿«,Í`°?o,­*­W®2 ́*ÂTō¿ÆlÍWoЮ « ° · °?aÔ« ́* ́IÝ «6­W3 ®?́~́*3Ô2 ̄†3 ¿°?« ̄›ÍWa Ò1⁄41⁄2T®2Í ¬q®?®? ̄wÍW·†® °?aÔ« ́* ́*®2́?±TÍW·†®?̄1⁄43 ÍÛ»e«?Ò 1⁄2T® ·¿«,­ Æ%®2­~ o,­BÍW·†®̄ ®2ÍA¬ o ­WÀeÍWoNa ® «,­W ̄1⁄4ÍW· ®™­ «,­W®2­B°?aÔ« ́* ́*®? ́I3Ô ̄ ́*o »N® ° « ́*®2 ́?̧Q %3⁄4 ° ·  ­ « ° ÍW3Ô°?« aQ® %ÂT®2­W3 ®? ̄†°?®„Æ%o›®? ́Û ̄†o Í~»e«6ÍW° ·lÍW·†® ?o,­W®2ÍW3 ° « aQ­W®? ́*3⁄4†a ÍW ́4¬~·†3 ° ·1⁄4 ́*·†o ¬ ÍW·¿«6Í 1⁄4Ü.Ó$́v«,† ­Wo %3 »e«,ÍW® «?қ®2 ́*3:«,̄ E ­Wo 1⁄2†« 1⁄2 3Ôa 3 ÍW3 ®? ́ =3 ̄†Æ ®?ÂT®? ̄†Æ ®? ̄›Í„o, ÍW· †®1⁄4Â%­W3Ôo,­°2a:«, ́* ́Â%­Wo 1⁄2¿«,1⁄2†3 a 3 ÍW3Ô®2 ́ ̧ Ú 3⁄4 ­e3 ̄ ÝG®? ́ ÍW3 2G«6ÍW3 o ̄ o ÍW· ®  ­Wo 1⁄2 a ®?» ́*·†o ¬~́ ÍW·¿«,͋ÍW·†®ÛÆ%3 1® ­W®? ̄†°?®~1⁄2T®2Í ¬q®?®2 ̄ÐÍW·†®4ÍW·†®?o,­W®2ÍW3 ° « a1«, ̄¿ÆÐÂ%­ « °2ÍW3 ° «,a†­W®? ́*3⁄4†a ÍẂEaÔ3 ®?́E¬~3 ÍW· ÍW·†®„« ́* ́*3⁄4 »N ÍW3 o ̄́I»e« Æ ® 3̄ÍW· ®„ÍW· ®?o ­*Ò « °2°?3⁄4 ­ «6ÍW®`®2 ́ ÍW3Ô»e«6ÍW3 o ̄ o «?қ®2 ́*3:«,̄ E ­Wo 1⁄2¿« 1⁄2 3 aÔ3 ÍW3 ®?́„­W® ›3⁄4†3 ­W®?́„ÍW·†®N ̄ ®2ÍA¬ o ­WÀ ÍWo 1⁄2T®laÔ«6­W2 ®e®? ̄†o 3⁄4†2 ·7±.Í*­ «,3 ̄ 3Ô ̄ 2wÍWo1⁄4°2o ̄ ÝG®2­W2 ®ÐÍWo1⁄4«2 aÔo 1⁄2¿«,a »N3 ̄†3 »v3⁄4 »1⁄4±¿3 ̄ †̄†3 ÍW®vÍ*­ «,3Ô ̄ 3 ̄†2 Ɔ«6Í «%±.« ̄†Æ ÍW·†® °?aÔ« ́*́e ­Wo 1⁄2¿«,1⁄2†3 a 3 ÍW3 ®? ́No, BÍW· ®wÍW®? ́ Íé*®2ÍvÍWo 1⁄2T®°2o ­*­W®?°2ÍWa Òt­W®?Â%­W®? ́*®?̄›ÍW® Æt3Ô ̄ ÍW·†®eÍ*­ « 3 ̄†3 ̄†2 ́*®2Í ̧ %ÂT®?°23 †° « a a қ±LÍW·†®w ­Wo 1⁄2†a ®?» °?« ̄ o, ;ÍW®? ̄1⁄2T®Í*­ «,°?® ƃÍWo ÍW· ® =« ° ÍvÍW·¿«6Í`® N°?3 ®? ̄›Íe 1⁄4Ü7Ó Í*­ « 3̄†3 ̄ 2 »N®?° ·†« ̄ 3Ô ́*»N ́„a ® « ƃÍWo ́*3⁄4 1⁄2 !o Â%ÍW3 »e« a ́*o a 3⁄4 ÍW3Ôo ̄† ́™ ;o ­¤»No ́ Í Â%­ « °2ÍW3 ° «,a  ­Wo 1⁄2†a ®?»Ń? ̧QÌĀ ÍW·†3 ́ ° ·¿« Â%ÍW®2­ ±Q¬q®NÆ%®?»No ̄ ́ Í*­ «,ÍW®vÍW· ®  ­Wo 1⁄2 a ®?»1⁄4±TÆ%3Ố*°23⁄4† ́* ́™ÂTo ́* ́*3Ô1⁄2 a ®v»N® 2ÍW· o Ǽ~ o ­ « a a ®?Ý 3:«6ÍW3 ̄†23 Í ±1«, ̄¿Æ 3 ̄›Í*­Wo Æ 3⁄4†°2®¤̄ ®2¬ ·†®?3⁄4%­W3Ô ́ ÍW3 °? ́$ÍW·¿«6ÍE«,­W®~́*·†o ¬~ ̄ÐÍWo ÂT®2­* ;o ­W» ¬q®?a a†o ̄e«™́W« »N aÔ® É Ä4 g°2a:«, ́* ́*3 ¿° «6ÍW3Ôo ̄  ­Wo 1⁄2 a ®?»1⁄4̧GÅq· ®I· ®?3⁄4 ­W3 ́ ÍW3 °? ́E»e« Òv« a ́*o„1⁄2T®~3⁄4† ́*® ÆN«, ́q« ́*3 »N†a ®~»N® « ̄́$o 7« Æ61 3⁄4† ́ ÍW3 ̄ 2 o ­Û3⁄4 ̄†® ›3⁄4¿«,a1»N3Ố*°2a:«, ́*́*3 ¿° «6ÍW3Ôo ̄ °?o ́ ÍW ́? ̧ !"#$%&'()*+*,-#&. '/! ’ ‡ œ Œ • 017ˆ ˆ, • œ ¢› t‡ œ ˆ,¢ Ž ˆ,‡ £ ,Œ §;§ 23ƒ‡ œ Œ ‡45 tŠ 67I•NŒ 89&9QŽ ¢ :;T£ 2< 1⁄4Œ ‡ ˆ1⁄4“™Œ :3%ˆ • £ Œ  =>?@&ABCDEFG H IJABH I 91Ž2¢ 1 QŒ 1 £;§:£:‡ £;ˆ •B œ ˆ  ‡ œ ˆv—1ˆ • £:Ž2ˆ — 1ˆ6‡WB¢%Ž2š1⁄4¢%”T‡ 91”T‡2• Œ%Ž ˆ KLA MNO Œ Q— • P †”QŒ%Ž ˆ — QAˆ Ž Ž2¢%ŽÛ¢%Ž 6Ž2¢ •2• QAˆ  ‡2Ž ¢ 9 3 6¢ • ‡(R ”1Q6‡ £;¢%Q•¤Œ Ž2ˆÐ”Q• ˆ —ST U ‘6V V %‘ 6VWXT‘ YVWZ ¿‘[X8\ T‘ [X Z¿‘-X U ‘[X8]T‘ [X8T̂‘ -\ XW_ ¥ ¡ œ £ • Ž2ˆ • ”1§Ô‡eŽ2ˆ,§;£;ˆ •N¢ ÊŒb¿” &<`17ˆ Že¢8R „Œ • • ”&< 9T‡ £;¢%7• 0R ¢%ŽeŒ ,6”QŽ2Œ ‡ ˆ ˆ • ‡ £2< Œ›‡2£:¢ Ya L‡ œ ˆ 1ˆ6‡WB¢%Ž2š Î Ü7®?®¤ B3 a ®? ́B3 ́I« a ́*oN¬~3 ÍW·1⁄4ÍW· ® ÌĀ ́ ÍW3 ÍW3⁄4%ÍW®™ o,­ Ã Æ Ý « ̄†°2® Æ1⁄4Ä o »N†3⁄4 ÍW® ­™ ›ÍW3⁄4¿Æ%3Ô®2 ́?±¿ B ̄ 3 ÝG®2­W ́*3 ÍAÒlo 1⁄4«,­*Ò%aÔ« ̄¿Æ1±TÄ o a aÔ®22 ®¤ÓL«,­WÀT±T ?b Þ Ö ß6Ï›Þ ̧ Î ÎcdBe:eWf6ghi:i j:j:jYkJl8m8n:opkJl q[kJlBm8n-kFnWr s i !  " # $ # # # % & ' ( $ )*+ ,   # # ( ! . / / + *+ (*0 + ' ( & 1 ! # 2 3 4  ,*0 *+ $ !.5"#6 78 / $ + % 9: ; < * *2 9 + 1 ( % => & ' &( )*+,   # # ( ? *+ + # ,./ + % " + ' 0 123&4@3"5 9:A +8, + $ 1*+ 6 " ; /7 *+ ' ( 8!.5 9#.*2 / % * 9 B C + 9 ( D E*+ , / F: + ! ? 2 ' + * F *2 / ( G H 2 B87 D / : * ' I 3 4 F J ; + D F # .K 6 # ( < ? ( * *+ , >89 # , % *+ , % = 2 :A + G*2 / ( E + 1 + > " & L D 9:A ? K *0 # 6 " "*+ 6 " *2 / ( @0 AK$BC 4 M$ , K # :A *+ , D7 : *+ 9 # , E F ? , G *2 6 ? L & + L*+ , 3N4 H0 I&4 # :> O + 8 ; 9 *+ , : # 9: J K / # # D & L G * # *+ ":6 =L # , M@NP L 6 :A + A*+ , % Q >*+ , R S + R # # OP 3 TJ * # Q K *+ A 9 U + # + ( % 9 # , G E=> & ( / Q 8 ( )R+ 9 # # ( ) * *2 9 $ , 9 ! / * ( % 9 # , S & < > " K ( / ( TN? 3 3 K H89 G J  " *+ , : *+ 9 # , .*2 ' UN ? > ,./ ? # G *2 C PFP 3 =A & V!G , ' W0XY 4C 78 ! K " * *2 9 & / 9 # , V )=> & ( / H ( )R+ Z # # ( L G * *+ :7 # , 3 W *  + K *2 / ( 9: [ & & K* *2 # , : # " 2 K K ( # : #   3 \ ]W^_à]UbcdFe f W 1 / (*0 X K *2 ( & ; * YK ? , 1 # +8J 9 # Z % ? , " + ' ( g 3 !G 89 9 ? , ) + ) / G ./ " + ' [ ) 8 98J K < G 8 ? 9: # *2 3 W U " 2 K B R*+ K + B /  YJ # G 6 *+ # : * + G ? h ( G "*2 i O H*2 i , / \ 3 jkl m n>o&pqr o skt u v pqw9x ] " + K ̂ *+ :6 # . 9 ( G *+ , Ô *2 . * *2 K $ *2 8   # # U , V V " *2 3U_ K + K + ' ,7 *2 , :6 # 9 ( 8 ? , ( * / "y9zK{| }hN ? :?PU~zK{ | }€N? , cP ‚ƒUzK{ | }€N„… PO: ƒUzK{ | } N„… P ~a†‡ˆ ‰2Š ‹XŒ  ‰2Žk F‘ $ …< > 9 / + < # ,. $ $z | } A O:6 # i 2 :A + + < /7 ;’ ! L & "“ O , ”̀ # L & "“€† 1 3 T *+ E:A + [ ̀ a , L K / O 9 / + • D*2 , / K ( 1 * 9 O [ .O *2 ( < 9 ; ? K 9 / + + , # H ( * *+ , B B ' ‡N P 3 3 K K + ' L U " J U O – / + B # L*2 hP,y —V˜"™ š Œ › b9œ  ž ƒ;z { | } N:…ŸP œ Œ D› žŸ¡ ~¢£ b ¤&¥ ¦•§$ ̈ N 1 P c)d e f g h,i[j'k+g/kFlmn o(i n,i f/k pSq r g h,i f s h'k2t(i[u e(n'svl-j i f i0j g h'iCu0k sviQe w'j,l X© i f i n9g u p-k svs`x f e(y'k y'lmpmlzg lmi s yKi gM{Ri i n g h,iAgvf/k lmn'lmn,oLk nJjLg i s@gEsvi g s |,i(q o,q a } « ¬ ?q ~  nJk+nJj i g k+pSq ‡a },¬>h'k0t7i&k pmsve.x,f i svi n g i j%k nVk+pmo(e f lzg h,€ f i p-k2g i0j g eG'n,i ­ 9Jk pUx f l-e+f u p#k+svs x f e(y 'k y'lmpmlzg lmi s q ® Ae0{Ri t7i f0|Fg h'i lmf"k pmo(e+f lzg h'€‚k+l-€Os e(n,pmƒ g e;lm€Ox f e+t(iHu e(n t7i f o i n'u iHsvxKi i0j q  j'j lmg lme nJk pmpzƒ9| g h'i lzf6k pmo e f lzg h'€ lmsUe(n'pzƒ w e+fEg„{Re!u p#k+svs6x f e(y,p-i €OsUk nJj yJk+g u/h ,x j'k2g i(q ! " # $ % & ' ( ) ' * + ,) ! * .' ' * / 0. 1 0 2 3 0.2 4 1 ' 5 , 6  0 & 7 0 1' 3 8 9 : ; * / , : 61 ;.0 * ! < ) : = " < #$ 1' 0 " ) > 3 = % " &' % ? 10 ) ( < ? ) ( ) / ? & * < #$ )10 0 @ ) 0.5 ; +, ' ) *. -+ 5 @ A 10 & ) 0 8 . + ) 1 ! B * / *.@ = & !1 C.' < ! 0.2 ,D .'10 / 0 @ * #$ 10 01 2 3 4 5 6789 :; <= 3 > ?@A B C EDEFG< HIJ : K = L E + ) 0 $ ' .4  8 9 F & ' ( / = #$ )10 & .4 > < D1 M DN : O = # 4 & ' ( ) P < ) G H = & 1 C.' B ) 2 .I ! & 0 ; ' ) & ' + N > $1Q ; J ) 0. : R ' & ) S ' D 0 D ' & & @ H 8 K2 & ' .' T 0 ' *. ; , I D, .' * 4 LM U VWJXQYUZ8[\]T^8_`Y ^8_* ' D 0 D ' & & @ ' D ( a:b < G ' c ' & ' I I ) ) d L # = e ; D c 0 . ! & .21 Q ' & ' ,) f 0 % !.' P < N > ' + 10 ' 3 0 O c ' g 0 0 & A @ G.' 01 h f= 8 8 F ' % * " " T ; / = = 3 & ' % # g % ! ; 6 g 2 4 8 9 0. B c ' P * 1' & ! ! G & ' c 4 U c ' , L i < ) j 4 ).' k * ! 4 U a ; G c ) & $ * ; # Q ' l 0 ? & ' 4 0.H ' ' ) .' T 0 ' *.L D ,) + ' 2.210 P i < ) j 4 ; 3 L & 3 : # B T D 8 9 * < & ! ' B & 0 B 1' ? c ?1 *.+ + / > B.' ! D C & ' ; # @ 3 # = $ % %1 #m M ne o"p o N : q = res o stM G 5 ) 0